Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

How do I create a list of modified files programmatically using linux command line tools? I'm not interested in the difference in any particular file (delta, patch). I just want to have a list of new or modified files comparing to previous product release. So that I can publish a new product update.

update: diff -qr doesn't produce very convinient output. The output of diff -qr also needs to be processed. Is there any better way?

share|improve this question

migrated from stackoverflow.com Oct 3 '11 at 13:24

This question came from our site for professional and enthusiast programmers.

    
what's an example of "convenient" output? – frogstarr78 Oct 4 '11 at 4:07

11 Answers 11

up vote 6 down vote accepted

I`ve got a simple approach for this: Use the rsync-preview mode:

rsync -aHSvn --delete old_dir/ new-dir/

The files that are shown as "to be deleted" by that command will be the "new" files. The others that are to be transferred have changed in some way. See the rsync-man-page for further details.

share|improve this answer

You can use the diff toool: see the options -q and -r

-q  --brief
Output only whether files differ.

-r  --recursive
Recursively compare any subdirectories found.

Example:

diff -qr dir1 dir2
share|improve this answer
    
Absolutely awful and unreadable output, cluttered with nonsense information saying Only in which does appears even if the directories are ideal copies. I was needed to compare changes against an old revision, and end up downloading whole revision into a separate directory, and using standard SVN tools to compare. That seems the only way to go… – Hi-Angel Jun 20 at 6:43

The diffutils package includes a lsdiff tool. Just pass the output of diff -u to lsdiff:

diff -u --other-diff-options path1 path2 | lsdiff
share|improve this answer
    
Good suggestion, thank you. Was in the patchutils package for me (CentOS 5.x). – Steve Kehlet Oct 18 '12 at 0:54
    
Yep, patchutils package for Ubuntu/Debian, too. – artfulrobot Mar 6 '14 at 12:56

I would just touch a file at the time of each update, and then you can find files that were modified since then with find /tree/location -newer /last/update/file -print

share|improve this answer

To take only the name of files that they changed, I use this command:

diff -r dirt1 dir2 --brief | sed 's/^Only in \([^:]*\): /\1\//' | sed 's/^Files \(.*\) and .* differ/\1/'

If need to exclude some files as object files or library files, you could use:

diff -r dirt1 dir2 --brief --exclude "*.o" --exclude "*.a" | sed 's/^Only in \([^:]*\): /\1\//' | sed 's/^Files \(.*\) and .* differ/\1/'
share|improve this answer

You should get desired result using:

diff -r --brief dir1/ dir2/
share|improve this answer

This might do the trick:

compare_dirs()
{
    # Shows which files and directories exist in one directory but not both
    if [ $# -ne 2 ]
    then
        echo "Usage: compare_dirs dir1 dir2" >&2
        return 2
    fi
    for path
    do
        if [ ! -d "$path" ]
        then
            echo "Not a directory: $path" >&2
            return 1
        fi
    done
    comm -3 \
        <(cd -- "$1" && find . -printf '%P\0' | sort -z | quote_shell) \
        <(cd -- "$2" && find . -printf '%P\0' | sort -z | quote_shell)
}
share|improve this answer

Normally you put the files into some kind of version control system like SubVersion or git, since those can do this for you out of the box.

But you could do a quick script with a for loop on dir1 and then compare every file with the one in dir2. The for loop can look at the exit code from diff to know if the files was different.

Maybe something like this:

for f in `(cd dir1 ; find .)`
do 
  diff $f ../dir2/$f
  if [ "$?" == "0" ]
  then 
    echo same
  else 
    echo diff: $f
  fi
done

Note: Script is not tested, so the above example is "bash inspired pseudocode"...


Let's take another go but with git

Create some example files to play with

mkdir -p dir1/test1/test11
mkdir -p dir1/test1/test12
mkdir -p dir1/test1/test13
echo "Test1" >> dir1/test1/test11/t1.txt
echo "Test2" >> dir1/test1/test12/t2.txt
echo "Test3" >> dir1/test1/test13/t3.txt

#And a dir to work in
mkdir gitdir

Then enter the dir and import dir1

cd gitdir/
git init .
cp -r ../dir1/* .
git add .
git commit -m 'dir1'

Go out and modify dir1 (so it becomes your dir2)

cd ..
echo "Test2" > dir1/test1/test11/t1.txt

Then go into the git dir and import the new dir

cd gitdir/
cp -r ../dir1/* .

Now ask git what has change (with the status command)

git status -s

The output is a list with the changes, that looks like this:

 M test1/test11/t1.txt
share|improve this answer

Maybe you'd be happier with something different. Try git.

Do this as an example:

mkdir a
cd a
git init
touch b
git add . && git commit -m "Empty file"
git status
echo c >> b
git status
git add . && git commit -m "Full file"
git status

git will track your files for you. The command git status will show you all the files that have been modified since the last commit.

share|improve this answer

To create a list of new or modified files programmatically the best solution I could come up with is using rsync, sort, and uniq:

(rsync -rcn --out-format="%n" old/ new/ && rsync -rcn --out-format="%n" new/ old/) | sort | uniq

Let me explain with this example: we want to compare two dokuwiki releases to see which files were changed and which ones were newly created.

We fetch the tars with wget and extract them into the directories old/ and new/:

wget http://download.dokuwiki.org/src/dokuwiki/dokuwiki-2014-09-29d.tgz
wget http://download.dokuwiki.org/src/dokuwiki/dokuwiki-2014-09-29.tgz
mkdir old && tar xzf dokuwiki-2014-09-29.tgz -C old --strip-components=1
mkdir new && tar xzf dokuwiki-2014-09-29d.tgz -C new --strip-components=1

Running rsync one way might miss newly created files as the comparison of rsync and diff shows here:

rsync -rcn --out-format="%n" old/ new/

yields the following output:

VERSION
doku.php
conf/mime.conf
inc/auth.php
inc/lang/no/lang.php
lib/plugins/acl/remote.php
lib/plugins/authplain/auth.php
lib/plugins/usermanager/admin.php

Running rsync only in one direction misses the newly created files and the other way round would miss deleted files, compare the output of diff:

diff -qr old/ new/

yields the following output:

Files old/VERSION and new/VERSION differ
Files old/conf/mime.conf and new/conf/mime.conf differ
Only in new/data/pages: playground
Files old/doku.php and new/doku.php differ
Files old/inc/auth.php and new/inc/auth.php differ
Files old/inc/lang/no/lang.php and new/inc/lang/no/lang.php differ
Files old/lib/plugins/acl/remote.php and new/lib/plugins/acl/remote.php differ
Files old/lib/plugins/authplain/auth.php and new/lib/plugins/authplain/auth.php differ
Files old/lib/plugins/usermanager/admin.php and new/lib/plugins/usermanager/admin.php differ

Running rsync both ways and sorting the output to remove duplicates reveals that the directory data/pages/playground/ and the file data/pages/playground/playground.txt were missed initially:

(rsync -rcn --out-format="%n" old/ new/ && rsync -rcn --out-format="%n" new/ old/) | sort | uniq

yields the following output:

VERSION
conf/mime.conf
data/pages/playground/
data/pages/playground/playground.txt
doku.php
inc/auth.php
inc/lang/no/lang.php
lib/plugins/acl/remote.php
lib/plugins/authplain/auth.php
lib/plugins/usermanager/admin.php

rsync is run with theses arguments:

  • -r to "recurse into directories",
  • -c to also compare files of identical size and only "skip based on checksum, not mod-time & size",
  • -n to "perform a trial run with no changes made", and
  • --out-format="%n" to "output updates using the specified FORMAT", which is "%n" here for the file name only

The output (list of files) of rsync in both directions is combined and sorted using sort, and this sorted list is then condensed by removing all duplicates with uniq

share|improve this answer

This is similar to rsync: show when newer file on destination is to be overwritten (asked later, though not a duplicate).

As indicated in the question, "diff -q -r" may require some processing to be useful. The question did not specify the form of the output; the answers give different types of reports.

rsync is a useful tool for this purpose because it is much faster than diff. However the solution suggested by @nils is far more verbose (and lists more files) than the actual differences between the old/new directory trees. For example, comparing that against the script which I wrote for that answer, and running on the same data,

  • @nils answer produces 605 lines (apparently because it includes directory changes),
  • "diff -q -r" produces 352 lines after running for several minutes, and
  • my script shows 252 lines (actual files changed, added or deleted)

To make diff properly account for new files, you need the -N option as well (which I do not see in any of the suggested answers). However, since it is far slower (orders of magnitude) than rsync, improving the latter's output seems the way to go.

Further reading

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.