Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Main question:

I am writing a script to perform a backup. I make a list of files in the following way:

LISTOFFILES=$(find ~ \( -name '*.[pP][dD][fF]' -o -name '*.[oO][dD][tT]' \))

The variable LISTOFFILES stores more or less something like:

/home/user/file1.pdf /home/user/file2.odt /home/user/Python Tutorial.pdf

My next step is:

tar cvf backup.tar $LISTOFFILES

Bash response is:

/home/user/file1.pdf
/home/user/file2.odt
/home/user/Python: Cannot stat: No such file or directory
tar: Tutorial.pdf

I understand that Bash assumes that Python and Tutorial.pdf are different files.

Is there a way to do a list of files with backslash included with find or should I change to use ls with pipes?

I'd like read your tips and advices.

My other question:

The answer to this question is not really important, but one tip could help

How to list files only with the file name without the file path?

For example: file instead of /home/user/file.

I've trying with find command with prune and path options, but I have not succeeded.

share|improve this question

3 Answers 3

up vote 6 down vote accepted

As for your main question, you'll need to take a different approach. Your method is putting all of the files into a single string in $LISTOFFILES. Then when you go to access it, you're not using quotes which causes it to split on spaces and give you the result you're seeing.

The simplest way to get the result you want is the following:

find ~ \( -name '*.[pP][dD][fF]' -o -name '*.[oO][dD][tT]' \) -print0 | xargs -0 tar cvf backup.tar

What we did there was to run find, and separate each result with a NULL char (the -print0 part). The output of find is then piped to xargs which reads in a list of files separated by the NULL char (the -0 argument tells xargs to do this), and pass that list of files as arguments to 'tar'.

For your second question: If you want to get just filenames only from the find command, use printf

find /path/to -printf '%f\n'
file.txt

If you mean in general, not just with find, it's easiest to use basename.

# basename /path/to/file.txt
file.txt

basename is designed to do exactly that, and nothing else.

share|improve this answer
1  
...or, if you don't want to spawn a new process with basename you could use parameter expansion. –  jasonwryan Nov 30 '11 at 6:28
    
Yes, was going for simple over efficient :-). There are also issues with that first command like when you get to large number of files as you can exceed the max number of arguments (in addition to spawning an extra proc for xargs). Can do a while read loop, but much more complex. –  Patrick Nov 30 '11 at 15:10
    
Thanks @Patrick and jasonwryan –  user12287 Nov 30 '11 at 20:43

Why your attempt failed and how to fix it

You can accumulate a list of file names returned by find, but only if there are no newlines in file names, and you need to take precautions.

When you write an unquoted command substitution or variable expansion, like $LISTOFFILES here, the shell performs word splitting and globbing (wildcard expansion) on the result.

Rule: always put double quotes around command substitutions and variable expansions.
Exception: if you understand why you need to leave off the double quotes, and why it's safe to do so.

Here, you need to leave off the double quotes, because word splitting is what splits the string into newline-separated bits. But by default it also splits at other whitespace characters. So you need to make that safe, and turn off globbing as well. You accomplish the former by setting the IFS variable to a single newline and the latter by invoking set -f.

IFS='
'
set -f
tar cvf backup.tar $LISTOFFILES

Note that there are no backslashes involved. Backslashes are used when you input filenames directly, not when they're produced by commands like find.

The usual way to use find

Rather than parse the output of find, you should make it invoke the command you want to run on the files, through the -exec action. This takes care of the case where there are so many files that the maximum command line length is exceeded: the program is invoked as many timed as necessary.

Here, this is a little difficult because you can't invoke tar -c multiple times (it would start over from a new empty archive at each invocation). But see Tar up all PDFs in a directory, retaining directory structure

Easier methods

If there aren't too many files, and you're running ksh93 or bash ≥4 or zsh, then use the ** wildcard to recurse into subdirectories. You need to set some shell options first:

set -o globstar                  # on ksh93
shopt -s globstar -s extglob     # on bash 4
setopt ksh_glob                  # on zsh
tar cvf backup.tar ~/**/*.@(PDF|pdf|ODT|odt)

In zsh, with no particular settings, you can write that pattern as ~/**/*.(PDF|pdf|ODT|odt). After setopt extglob, you can write it as ~/**/(#i)*.(pdf|odt).

Or you can use pax:

pax -w -x ustar -s '/\.pdf$/&/' -s '/\.PDF$/&/' -s '/\.odt$/&/' -s '/\.ODT$/&/' -s '/.*//' ~ >backup.tar

If you don't want the path prefix

There are ways with tar and pax, but the simplest approach is to change to the directory you want to archive first.

( IFS='
'; set -f; cd ~ && tar cvf /path/to/backup.tar $(find . …) )
( cd ~ && pax -w … . ) >backup.tar
share|improve this answer

First question:

The answer to your first question is quotes.

Instead of doing -> tar cvf backup.tar $LISTOFFILES

Do this -> tar cvf backup.tar "$LISTOFFILES"

Let's take a look at an example (I took one of your file for this demo, even though the file does not exist on my computer, look at the error's I am getting with and without quotes) -

[jaypal:~] file="/home/user/Python Tutorial.pdf"

[jaypal:~] ls $file
ls: /home/user/Python: No such file or directory
ls: Tutorial.pdf: No such file or directory

Error appeared for two files /home/user/Python and Tutorial.pdf. Let's put a quote to our variable and see what we get.

[jaypal:~] ls "$file"
ls: /home/user/Python Tutorial.pdf: No such file or directory

Error appeared only once for file /home/user/Python Tutorial.pdf

Second question:

To prune the /path/to/file there are several ways. My favourite would be awk but I'll show you one for sed too.

sed method:

[jaypal:~/Temp] echo "$filename"
/home/user/file1.pdf /home/user/file2.odt /home/user/Python Tutorial.pdf

[jaypal:~/Temp] echo "$filename" | sed 's@/home/user/@@g'
file1.pdf file2.odt Python Tutorial.pdf

You can include it in your script like this -

LISTOFFILES=$(find ~ \( -name '*.[pP][dD][fF]' -o -name '*.[oO][dD][tT]' \) | sed 's@/home/user/@@g' )

Thing to remember here is we are doing substitution. We are substituting /home/user/ to nothing. If you are finding files from more than one directories then you can either make tons and tons of substitutions (yikes!) or do the awk way.

awk method:

awk -F"/" '{print $NF}'

Yep, thats it. Would work for files you find under any directory or their sub-directories.

So your script could have this in the following way -

LISTOFFILES=$(find ~ \( -name '*.[pP][dD][fF]' -o -name '*.[oO][dD][tT]' \) | awk -F"/" '{print $NF}')

Other Tips and Tricks:

Here are some random tips that I think may be useful to you -

To find file names insensitively, use iname instead of what you have right now. Check this out -

[jaypal:~/Temp] find . -name "j*.txt"
./jP.txt

[jaypal:~/Temp] find . -iname "j*.txt"
./jj.TXT
./jP.txt

Other ways of pruning the /path/to/file is by using basename. I will give an example and you can incorporate it in your script as you feel comfortable.

[jaypal:~/Temp] filename="/usr/bin/java.pdf"

[jaypal:~/Temp] echo "$filename"
/usr/bin/java.pdf

[jaypal:~/Temp] basename "$filename"
java.pdf

Hope this helps!

share|improve this answer
    
Your quote solution wont work because all the files will be joined together as a single argument. The only way to do that with shell variables would be to use an array which cant work (reliably) with the approach originally used. –  Patrick Nov 30 '11 at 6:18
    
I tried on my computer. Worked ok. [jaypal:~/Temp] file=$(find . -name "*p*" | awk -F"/" '{print $NF}') [jaypal:~/Temp] for i in "$file"; do echo "$i"; done backup split.pl XDR_Split.pl gtpparser.sh all files appear on new lines. –  jaypal Nov 30 '11 at 6:24
1  
They appeared on separate lines because find printed them out on separate lines. Change your echo to echo "<$i>" and watch it fail spectacularly :-) –  Patrick Nov 30 '11 at 14:44
    
Thanks @Jaypal I'll play with <code>awk</code> and <code>sed</code> when I'll have time. About the <code>iname</code> option, I am writing a script that should work in Solaris and Linux, <code>iname</code> is not available in Solaris. –  user12287 Nov 30 '11 at 20:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.