8

I have a base Linux system installed. I want to run a very large and complex third-party script that will make many changes to various parts of the entire system. These changes will include adding new files, modifying existing files, and removing existing files. Once the script is complete, I will have a modified system. The modifications will be broad and substantial.

I want some sort of report on all the files that have been added, removed, or modified and a line-by-line analysis of the modifications. Something like the output of a file diff tool would be great.

I assume I need to make some sort of snapshot before running the script and a second snapshot after. I assume then I would feed those snapshots into some sort of diff or diff-like tool.

Does anyone know what tools to use and how to diff an entire system?

I am using Virtualbox which does have a differencing images feature, though I don't know if I can adapt it to this purpose. Moreover, I'd prefer a more generic solution if possible.

  • Differencing images (aka snapshots) are not what you're looking for. Their goal is to produce a binary diff as small as possible, not a verbose text diff. – Dmitry Grigoryev Apr 22 '15 at 8:01
  • @DmitryGrigoryev That's the impression I got from the link. Thanks for confirming this for me. – user38810 Apr 22 '15 at 8:04
  • In fact what you want is quite easy but complicated to achieve. Given the resources I would install two system exactly the same. Enable filesystem audit on the system that will have the files modified. Get the report from audit to identify what was added/modfiry/delete. Then compare these files with the other installed system. – BitsOfNix Apr 22 '15 at 8:08
  • 1
    You can use rsync to make a snapshot and another run after the modifications to generate a report of modified files. The diffing part would probably be harder. – FloHimself Apr 22 '15 at 8:18
1

I think your idea is not far from a solution. To outline a possible way: I am using rsnapshot for backups. It creates a directory (backup-)structure of all or of a subset of your files with entry points of (e.g.) /backup/hourly.1/... and /backup/hourly.0/..., where each branch carries the whole data, but using (hard-)links for files where no changes have been done. Doing a recursive ls or find on both structures and comparing the (sorted, in case of find) output will show the missing files, and inspecting the link-count (in ls -l it would be the second column) will show new files (which have a link count 1). For details of changes in the files you can (for the identified files) use ordinary diff tools. As said this is an outline, will need some work to implement, and may have non-apparent quirks, so take that proposal with a grain of salt

0

I would probably do this using rsync as suggested in one of the comments. Rsync has a dry-run mode and checksum feature you could use to create an accurate report of what's changed.

Taking it further you could perhaps write a script that performs a diff on changed files, though you'd have to make it avoid binary files (not sure how)..

Or, you could take a totally different approach.. use git. So on your 'base' system, perform a git init in /, then after you've run the third party thing you could just do:

git status > /tmp/changed-files.txt
git diff > /tmp/changes.txt

using git would give you quite a bit of flexibility. Having said that it may struggle over an entire system and might get confused about /proc etc..

0

There is something called libguestfs that you can use to perform the diff. That's what I have used in the past and I think now that operation might even be built in so you won't have to use the guestfish shell.

Another option is to run your operations in a docker container and then export and diff the resulting file trees as tar files.

Be aware though that if the script changes system settings by writing to virtual files then those changes will not show up in the file tree and you need something else to capture those changes. An example of something like that is iptable rules. You'll need custom logic to capture those changes.

  • Except if the rule is persistent then it must be stored somewhere so it can be re-run/re-enabled after network comes up, etc. – ivanivan Nov 29 '17 at 19:46
0

The page at The Linux Cookbook shows a pre and post modification find-grep, followed by a diff. It's very simple, it might be a starting point, but your problem might be better addressed with the more sophisticated solutions noted.

This looks like it would only catch additions and deletions, not modifications ... cheers, drl

0

All of these answers are headed in the right direction and the same way. May as well throw my 8 bits in...

Install base system on virtualbox. After initial setup and your base is ready, boot the vm with some other live media. You can then tar up or otherwise export the file system without the volatile directories like /proc being involved.

Run your massive script, and repeat the export process.

Now you have the 2 filesystems available, you can use various comparison tools.

diff --brief -Nr /tree1 /tree2

Will give you a nice list of files that differ on stdout, and a list of files that aren't found in one tree or the other on stderr.

To find out what actual changes took place, you could parse the stdout output of files that differ and run a regular diff on them individually, redirecting output to a file. Or you could just examine the list and run diff on the files that particularly interest you.

You could combine all of this into one command, using git. Can be used without having a repo initialized, just point to any 2 directories. Gives a nice colored output, paginated through less.

git diff --no-index /tree1 /tree2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy