Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm trying to invoke a script with a list of filenames collected by find. Nothing special, just someting like this:

$ myscript `find . -name something.txt`

The problem is that some of the pathnames contain spaces, so they get broken up into two invalid names on argument expansion. Normally I would surround the names with quotes, but here they're inserted by the backquote expansion. I've tried filtering the output of find and surrounding each filename with quotes, but by the time bash sees them, it's too late to strip them and they are treated as part of the filename:

$ myscript `find . -name something.txt | sed 's/.*/"&"/'`
No such file or directory: '"./somedir/something.txt"'

Yes, that's the rules for how the command line is processed, but how do I get around it?

This is embarrassing but I'm failing to come up with the right approach. I finally figured out how to do it with xargs -0 -n 10000... but it's such an ugly hack that I still want to ask: How do I quote the results of backquote expansion, or achieve the same effect in another way?

Edit: I was confused about the fact that xargs does collect all arguments into a single argument list, unless it's told otherwise or system limits might be exceeded. Thanks to everyone for setting me straight! Others, keep this in mind as you read the accepted answer because it's not pointed out very directly.

I've accepted the answer, but my question remains: Isn't there some way to protect spaces in backtick (or $(...)) expansion? (Note that the accepted solution is a non-bash answer).

share|improve this question
    
I guess you'd need to change what does the shell use as filename separators (for example, by playing with the value of IFS, one possible way is IFS=", newline, "). But is there a need to execute the script over all the filenames? If not, consider using find itself to execute the script for each file. –  njsg Jan 19 at 23:30
    
Changing the IFS is a great idea, hadn't thought of it! Not practical for commandline usage, but still. :-) And yes, the goal is to pass all the arguments to the same invocation of my script. –  alexis Jan 20 at 1:30

4 Answers 4

up vote 3 down vote accepted

You could do the following using some implementations of find and xargs like this.

$ find . -type f -print0 | xargs -r0 ./myscript

or, standardly, just find:

$ find . -type f -exec ./myscript {} +

Example

Say I have the following sample directory.

$ tree
.
|-- dir1
|   `-- a\ file1.txt
|-- dir2
|   `-- a\ file2.txt
|-- dir3
|   `-- a\ file3.txt
`-- myscript

3 directories, 4 files

Now let's say I have this for ./myscript.

#!/bin/bash

for i in "$@"; do
    echo "file: $i"
done

Now when I run the following command.

$ find . -type f -print0 | xargs -r0 ./myscript 
file: ./dir2/a file2.txt
file: ./dir3/a file3.txt
file: ./dir1/a file1.txt
file: ./myscript

Or when I use the 2nd form like so:

$ find . -type f -exec ./myscript {} +
file: ./dir2/a file2.txt
file: ./dir3/a file3.txt
file: ./dir1/a file1.txt
file: ./myscript

Details

find + xargs

The above 2 methods, though looking different, are essentially the same. The first is taking the output from find, splitting it using NULLs (\0) via the -print0 switch to find. The xargs -0 is specifically designed to take input that's split using NULLs. That non-standard syntax was introduced by GNU find and xargs but is also found nowadays in a few others like most recent BSDs. The -r option is required to avoid calling myscript if find finds nothing with GNU find but not with BSDs.

NOTE: This entire approach hinges on the fact that you'll never pass a string that's exceedingly long. If it is, then a 2nd invocation of ./myscript will get kicked off with the remainder of subsequent results from find.

find with +

That's the standard way (though it was only added relatively recently (2005) to the GNU implementation of find). The ability to do what we're doing with xargs is literally built into find. So find will find a list of files and then pass that list as as many arguments as can fit to the command specified after -exec (note that {} can only be last just before + in this case), running the commands several times if needed.

Why no quoting?

In the first example we're taking a shortcut by completely avoiding the issues with the quoting, by using NULLs to separate the arguments. When xargs is given this list it's instructed to split on the NULLs effectively protecting our individual command atoms.

In the second example we're keeping the results internal to find and so it knows what each file atom is, and will guarantee to handle them appropriately, thereby avoiding the whoie business of quoting them.

Maximum size of command line?

This question comes up from time to time so as a bonus I'm adding it to this answer, mainly so I can find it in the future. You can use xargs to see what the environment's limit like so:

$ xargs --show-limits
Your environment variables take up 4791 bytes
POSIX upper limit on argument length (this system): 2090313
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2085522
Size of command buffer we are actually using: 131072
share|improve this answer
    
Thanks but I need to pass all the arguments to the same invocation of my script. That's in the problem description, but I guess I didn't make it clear that it's not incidental. –  alexis Jan 20 at 1:09
    
@alexis - read the answers again, they are passing all the arguments to a single call of your script. –  slm Jan 20 at 1:15
    
I'll be damned! I didn't know about the + argument to find (and you use + in prose too, so I missed your explanation the first time). But more to the point, I'd misunderstood what xargs does by default!!! In three decades of using Unix I've never had a use for it until now, but I thought I knew my toolbox... –  alexis Jan 20 at 1:27
    
@alexis - I figured you'd missed what we were saying. Yes xargs is a devil of a command. You have to read it and find's man pages many times over to grok what they can do. May of the switches are contra-positives of each other so that adds to the confusion. –  slm Jan 20 at 1:32
    
@alexis - also one more thing to add to the tool box, don't use the backquotes/backticks for running nested commands, use $(..) now instead. It automatically handles nesting of quotes etc. Backticks are being deprecated. –  slm Jan 20 at 1:34
find . -name something.txt -exec myscript {} +

In the above, find finds all the matching file names and provides them as arguments to myscript. This works with file names regardless of spaces or any other odd characters.

If all the file names fit on one line, then myscript is executed once. If the list is too long for the shell to handle, then find will run myscript multiple times as needed.

MORE: How many files fit on a command line? man find says that find builds it command lines "much the same way that xargs builds its". And, man xargs that the limits are system dependent and that you can determine them by running xargs --show-limits. (getconf ARG_MAX is also a possibility). On Linux, the limit is typically (but not always) around 2 million characters per command line.

share|improve this answer

A few addition to @slm's fine answer.

The limitation on the size of the arguments is on the execve(2) system call (actually, it's on the cumulative size of the argument and environment strings and pointers). If myscript is written in a language that your shell can interpret, then maybe you don't need to execute it, you could have your shell just interpret it without having to execute another interpreter.

If you run the script as:

(. myscript x y)

It's like:

myscript x y

Except that it's being interpreted by a child of the current shell, instead of executing it (which eventually involves executing sh (or whatever the she-bang line specifies if any) with even more arguments).

Now obviously, you can't use find -exec {} + with the . command, as . being a builtin command of the shell, it has to be executed by the shell, not by find.

With zsh, it's easy:

IFS=$'\0'
(. myscript $(find ... -print0))

Or:

(. myscript ${(ps:\0:)"$(find ... -print0)"}

Though with zsh, you wouldn't need find in the first place as most of its features are built into zsh globbing.

bash variables however cannot contain NUL characters, so you have to find another way. One way could be:

files=()
while IFS= read -rd '' -u3 file; do
  files+=("$file")
done 3< <(find ... -print0)
(. myscript "${files[@]}")

You might also use zsh-style recursive globbing with with globstar option in bash 4.0 and later:

shopt -s globstar failglob dotglob
(. myscript ./**/something.txt)

Note that ** followed symlinks to directories until it was fixed in bash 4.3. Also note that bash doesn't implement zsh globbing qualifiers so you won't get all the features of find there.

Another alternative would be to use GNU ls:

eval "files=(find ... -exec ls -d --quoting-style=shell-always {} +)"
(. myscript "${files[@]}")

The above methods can also be used if you want to make sure myscript is executed only once (failing if the argument list is too large). On recent versions of Linux, you can raise and even lift that limitation on the argument list with:

ulimit -s 1048576

(1GiB stack size, a quarter of which can be used for the arg+env list).

ulimit -s unlimited

(no limit)

share|improve this answer

In most systems, there is a limit on the length of a commandline passed to any program, using xargs or -exec command {} +. From man find:

-exec command {} +
      This  variant  of the -exec action runs the specified command on
      the selected files, but the command line is built  by  appending
      each  selected file name at the end; the total number of invoca‐
      tions of the command will  be  much  less  than  the  number  of
      matched  files.   The command line is built in much the same way
      that xargs builds its command lines.  Only one instance of  `{}'
      is  allowed  within the command.  The command is executed in the
      starting directory.

Invocations will be much less, but not guaranteed to be one. What you should do is read the NUL separated filenames in the script from stdin, possible based on a commandline argument -o -. I would do something like:

$ find . -name something.txt -print0 | myscript -0 -o -

and implement the option arguments to myscript accordingly.

share|improve this answer
    
Yes, the OS imposes a limit in the number/size of arguments that can be passed. On modern Linux systems this is (gigantic)(linux.die.net/man/2/execve) (1/4 of stack size, 0x7FFFFFFF arguments). AFAIK bash itself doesn't impose any limit. My lists are much smaller, and my problem was caused by misunderstanding or mis-remembering how xargs works. Your solution is indeed the most robust, but it's overkill in this case. –  alexis Jan 20 at 10:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.