Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have this very simple line in a bash script which executes successfully (i.e. producing the _data.tar file), except that it doesn't exclude the sub-directories it is told exclude via the --exclude option:

/bin/tar -cf /home/_data.tar  --exclude='/data/sub1/*'  --exclude='/data/sub2/*' --exclude='/data/sub3/*'  --exclude='/data/sub4/*'  --exclude='/data/sub5/*'  /data

Instead, it produces a _data.tar file that contains everything under /data, including the files in the subdirectories I wanted to exclude.

Any idea why? and how to fix this?

Update I implemented my observations based on the link provided in the first answer below (top level dir first, no whitespace after last exclude):

/bin/tar -cf /home/_data.tar  /data  --exclude='/data/sub1/*'  --exclude='/data/sub2/*'  --exclude='/data/sub3/*'  --exclude='/data/sub4/*'  --exclude='/data/sub5/*'

But that didn't help. All "excluded" sub-directories are present in the resulting _data.tar file.

This is puzzling. Whether this is a bug in current tar (GNU tar 1.23, on a CentOS 6.2, Linux 2.6.32) or "extreme sensitivity" of tar to whitespaces and other easy-to-miss typos, I consider this a bug. For now.

This is horrible: I tried the insight suggested below (no trailing /*) and it still doesn't work in the production script:

/bin/tar -cf /home/_data.tar  /data  --exclude='/data/sub1'  --exclude='/data/sub2'  --exclude='/data/sub3'  --exclude='/data/sub4'

I can't see any difference between what I tried and what @Richard Perrin tried, except for the quotes and 2 spaces instead of 1. I am going to try this (must wait for the nightly script to run as the directory to be backed up is huge) and report back.

/bin/tar -cf /home/_data.tar  /data --exclude=/data/sub1 --exclude=/data/sub2 --exclude=/data/sub3 --exclude=/data/sub4

I am beginning to think that all these tar --exclude sensitivities aren't tar's but something in my environment, but then what could that be?

It worked! The last variation tried (no single-quotes and single-space instead of double-space between the --excludes) tested working. Weird but accepting.

Unbelievable! It turns out that an older version of tar (1.15.1) would only exclude if the top-level dir is last on the command line. This is the exact opposite of how version 1.23 requires. FYI.

share|improve this question

6 Answers 6

up vote 12 down vote accepted

If you want to exclude an entire directory, your pattern should match that directory, not files within it. Use --exclude=/data/sub1 instead of --exclude='/data/sub1/*'

Be careful with quoting the patterns to protect them from shell expansion.

See this example, with trouble in the final invocation:

$ for i in 0 1 2; do mkdir -p /tmp/data/sub$i; echo foo > /tmp/data/sub$i/foo; done
$ find /tmp/data
/tmp/data
/tmp/data/sub2
/tmp/data/sub2/foo
/tmp/data/sub0
/tmp/data/sub0/foo
/tmp/data/sub1
/tmp/data/sub1/foo
$ tar -zvcf /tmp/_data.tar /tmp/data --exclude='/tmp/data/sub[1-2]'
tar: Removing leading `/' from member names
/tmp/data/
/tmp/data/sub0/
/tmp/data/sub0/foo
$ tar -zvcf /tmp/_data.tar /tmp/data --exclude=/tmp/data/sub[1-2]
tar: Removing leading `/' from member names
/tmp/data/
/tmp/data/sub0/
/tmp/data/sub0/foo
$ echo tar -zvcf /tmp/_data.tar /tmp/data --exclude=/tmp/data/sub[1-2]
tar -zvcf /tmp/_data.tar /tmp/data --exclude=/tmp/data/sub[1-2]
$ tar -zvcf /tmp/_data.tar /tmp/data --exclude /tmp/data/sub[1-2]
tar: Removing leading `/' from member names
/tmp/data/
/tmp/data/sub2/
/tmp/data/sub2/foo
/tmp/data/sub0/
/tmp/data/sub0/foo
/tmp/data/sub2/
tar: Removing leading `/' from hard link targets
/tmp/data/sub2/foo
$ echo tar -zvcf /tmp/_data.tar /tmp/data --exclude /tmp/data/sub[1-2]
tar -zvcf /tmp/_data.tar /tmp/data --exclude /tmp/data/sub1 /tmp/data/sub2
share|improve this answer
    
Thanks for the very focused and clear answer. Regarding your 1st point, I was trying to follow the tips in this LQ thread. I am not sure what I missed, but now that I read your 2nd point it may very well an absolute vs. relative path issue. I will try that and report back. +1 for now. –  ateiob Feb 29 '12 at 14:15
    
Another thing I noticed is the --exclude b (space instead of equal sign) vs. --exclude=b. Does this make any difference? (shouldn't IMHO) –  ateiob Feb 29 '12 at 14:26
1  
The equal sign can be essential to avoid shell expansion of unquoted patterns. If you have a space instead, then an unquoted pattern can be expanded by the shell into a single --exclude argument, and the remaining expansions give as files to add to the tar file. Your examples above all have '=' - if the script doesn't, and is missing single quotes, then that can the source of your problem. –  R Perrin Feb 29 '12 at 17:26
    
OK. I tested your example on my box and it works, even with multiple --exclude= on the same line. So the difference must be the idiotic /* that I appended to each sub-directory. I will test this tonight in the production script and report back. Another +1. –  ateiob Feb 29 '12 at 21:43

For excluding multiple files, try

--exclude=/data/{sub1,sub2,sub3,sub4}

This will save some code and headache. This is a global solution, for all kind of programs / options. If you also want to include the parent directory in your selection (in this case data), you have to include a trailing comma. E.g.:

umount /data/{sub1,sub2,}
share|improve this answer

Perhaps you can try the command with another option:

--wildcards

And check if it's running as intended.

share|improve this answer
    
See my update above. The last variation tried (no quotes, single space) works. I have no idea why. +1 for the idea. –  ateiob Mar 4 '12 at 3:53

It may be that your version of tar requires that the --exclude options have to be placed at the beginning of the tar command.

See: http://stackoverflow.com/q/984204

tar --exclude='./folder' --exclude='./upload/folder2' \
    -zcvf /backup/filename.tgz .

See: http://mandrivausers.org/index.php?/topic/8585-multiple-exclude-in-tar/

tar --exclude=<first> --exclude=<second> -cjf backupfile.bz2 /home/*

Alternative:

EXCLD='first second third'
tar -X <(for i in ${EXCLD}; do echo $i; done) -cjf backupfile.bz2 /home/*

Yet another tar command tip is from here:

tar cvfz myproject.tgz --exclude='path/dir_to_exclude1' \
                       --exclude='path/dir_to_exclude2' myproject
share|improve this answer
    
See my update above. The last variation tried (no quotes, single space) works. I have no idea why. +1 for the well thought answer + links. –  ateiob Mar 4 '12 at 3:51

A workaround may be to use a combination of find ... -prune and tar to exclude the specified directories.

On Mac OS X the --exclude option of GNU tar seems to work as it should though.

In the following test case the directories /private/var/log/asl and /private/var/log/DiagnosticMessages are to be excluded from a compressed archive of the /private/var/log directory.

# all successfully tested in Bash shell on Mac OS X (using gnutar and gfind)

# sudo port install findutils  # for gfind from MacPorts

sudo gnutar -czf ~/Desktop/varlog.tar.gz /private/var/log --exclude "/private/var/log/asl" --exclude "/private/var/log/DiagnosticMessages"

sudo gnutar -czf ~/Desktop/varlog.tar.gz  --exclude "/private/var/log/asl" --exclude "/private/var/log/DiagnosticMessages" /private/var/log

set -f # disable file name globbing
sudo gnutar -czf ~/Desktop/varlog.tar.gz  --exclude "/private/var/log/asl" --exclude "/private/var/log/Diagnostic*" /private/var/log

# combining GNU find and tar (on Mac OS X)

sudo gfind /private/var/log -xdev -type d \( -name "asl" -o -name "DiagnosticMessages" \) -prune -o -print0 | 
   sudo gnutar --null --no-recursion -czf ~/Desktop/varlog.tar.gz --files-from -

# exclude even more dirs
sudo gfind /private/var/log -xdev -type d \( -name "asl" -o -name "[Dacfks]*" \) -prune -o -print0 | 
    sudo gnutar --null --no-recursion -czf ~/Desktop/varlog.tar.gz --files-from -


# testing the compressed archive

gnutar -C ~/Desktop -xzf ~/Desktop/varlog.tar.gz

sudo gfind /private/var/log ~/Desktop/private \( -iname DiagnosticMessages -or -iname asl \)

sudo rm -rf ~/Desktop/varlog.tar.gz ~/Desktop/private
share|improve this answer
    
Thanks +1 for the suggestion. At this point I am still trying to understand why a well documented (and mature) feature isn't working in my script, nightly run by cron. –  ateiob Mar 2 '12 at 14:52

This link might be helpful. http://answers.google.com/answers/threadview/id/739467.html

Two immediate differences between the non-working line and some tips in the link:

  1. All excludes come after the top-level directory.
  2. Cannot have ANY spaces after the last --exclude.
share|improve this answer
    
Thanks. The answer by -MAK caught my attention and so far I have been able to spot the following differences between my non-working line and the following: 1. All excludes come after the top-level directory. 2. Cannot have ANY spaces after the last --exclude. I will test these insights and report back. +1 for now. –  ateiob Feb 27 '12 at 16:20
    
@ateiob If you figure it out, can you post an answer here or edit this one? We generally prefer not having answers that are just links elsewhere –  Michael Mrozek Feb 27 '12 at 16:22
    
@Michael Mrozek Absolutely. This is exactly what I wrote in my comment. :) –  ateiob Feb 27 '12 at 16:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.