Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Albert Einstein quote

Insanity: doing the same thing over and over again and expecting different results

Often-times, Linux drives me mad because I'm doing the same thing over and over again and getting different results from box to box. (See my previous question).

For me, the biggest area of confusion is taking over a machine that someone else has installed (as is the case when signing up with a web hosting company). You just don't know what you're dealing with.

Is there some kind of clever diff tool that I can run on an installation of Linux (Ubuntu) to give me a heads-up on how that machine has veered from the default installation?

i.e. Something that can show me a list of the commands that are going to behave surprisingly thus avoiding a trial and error approach.

share|improve this question
You won't get a “list of the commands that are going to behave surprisingly”. Software can't predict what you'll find surprising. –  Gilles May 21 '11 at 13:42
Thanks. These answers are giving me better understanding of how to model the pertinent areas of difference that need to be considered when taking over a machine that someone else has installed. So far we have differences in: packages installed, running services, configurations and file system type. I'm interested to know if that's it or is the list infinite? @Gilles - yes, with limited knowledge of linux, I find it hard to predict what I'd 'find surprising' myself as well! –  JW01 May 21 '11 at 15:43
The list is infinite. Not a Linux example, but a while ago, Visual Studio wouldn't show some dialog boxes on a particular machine (no error, just empty space where the controls should have been). It turned out to be due to having too many fonts installed. The moral of this story is that there will always be surprises lurking in corners. –  Gilles May 21 '11 at 15:51

5 Answers 5

up vote 4 down vote accepted

Whenever I have a good reference system and a misbehaving one, I try to compare them with vimdiff. What I compare varies with the problem, e.g.

1) When comparing servers at the package level, I create sorted lists of packages on each server, send the results to files and diff them, e.g.

On server1:

dpkg --get-selections|sort > server1_packages

On server2:

dpkg --get-selections|sort > server2_packages

Copy both files to the same machine and diff (or vimdiff) them.

2) Make a list of running services as in example 1

sysv-rc-conf --list|sort > server1_services

sysv-rc-conf --list|sort > server2_services

...etc., and vimdiff those.

3) If you are troubleshooting inconsistent configurations with Apache for example, make copies of the config files, and vimdiff those, etc.

share|improve this answer
Good summary of options. It is also a good idea to put files under /etc under version control to help you keep track of what is going on. etckeeper will do this for you. –  Faheem Mitha May 21 '11 at 12:57
I am getting "dpkg: failed to open package info file `/var/lib/dpkg/status' for reading: No such file or directory". How would this look on Centos with yum? PS Nevermind, found it "yum list installed" –  giorgio79 Dec 6 '13 at 9:02

Most of the differences between two installations of the same distribution will be in /etc. Copy the contents of /etc from one machine to a temporary directory on the other and run

diff -ru /etc /copy/of/other/etc

If you want to compare a machine with a default installation, get a copy from a fresh default installation (perhaps in a virtual machine).

If you can control a machine from the start, make sure to install etckeeper to keep /etc under version control. Then you'll be able to see precisely what's changed.

Of course, there'll be a lot of differences. If you don't yet have an idea of where to look, this is the wrong approach to investigate a particular difference in behavior. For example, let's say the command adduser behaves differently on two machines. Then the best approach is to look at what adduser is doing; run it on both machines and compare. If it has an option to tell it to be more verbose (adduser doesn't), use it. More radically, run the program under strace, e.g.

strace -s9999 -efile adduser …

to see what files adduser accesses.

share|improve this answer

To harken back to an earlier question, your nifty tool, which sounds like a good idea to me, but I've never heard of such a beast existing, needs to check things like filesystem type and sizes.

A real life example that happened to me in 2002: two people installed SuSE 7.3 systems on two adjacent rack-mounted servers. They put a lot of effort into getting the packages exactly the same on both servers. A short while later, we had problems with some software in development. It came down to ext3 filesystem on one server, Reiserfs on the other. A plain "ls" on Reiserfs gives file names in lexical order, but not on ext3. One program failed on out-of-order filenames.

share|improve this answer

For files, users, groups, packages, services etc I've used http://www.scriptrock.com; it's free for a few servers and gives you a nice visual comparison of the differences between the servers..very handy for drift

share|improve this answer

"...your nifty tool, which sounds like a good idea to me, but I've never heard of such a beast existing..."

The beast now exist :-) Check out http://metaforsoftware.com. It can do large scale diffs across multiple servers as well as over time. FD: I'm a co-founder.

share|improve this answer
+1 Looks very interesting. –  JW01 May 22 '13 at 15:21
Thanks @JW01! Our tool is in free beta right now and we plan to always have a free version. Hope it can help cure some inconsistency headaches :) –  jyang Jun 8 '13 at 4:55

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.