Having some trouble using "find -exec {} +"

Question

Disclaimer: I am a novice to Unix/Linux, but I am looking forward to learning! I have tried a search on this stackexchange and read the the man find, but I can't seem to figure this out.

I want to use the find ... -exec {} + command to recursively find all files with a particular file extension and run a command on the list of files. There are approximately 100k files that I need to convert. The command that I am running accepts the filename (or a list of filenames, eg f1 f2 f3) as a parameter, but I also need to specify additional parameters to run the command.

What I tried so far:

This works:

find . -iname "*.extension" -exec <command> {} <additional parameters> \;

This doesn't seem to work:

find . -iname "*.extension" -exec <command> {} <additional parameters> +

I get the error message, find: missing argument to '-exec'. I am guessing that I cannot specify additional parameters after the {}?

Some notes:

The command in question takes the filename as the first parameter, and then I need to designate some additional parameters, such as the output directory -o <outputDir> and the variables to extract from the files -v <var1,var2,...>.

I am running this on the terminal in Ubuntu 12.04, if that makes any difference.

Answer 1

find . -iname "*.extension" -exec sh -c '
  exec <command> "$@" <additional parameters>' sh {} +

See How does this find command using "find ... -exec sh -c '...' sh {} +" work? for details.

Answer 2

With the + it's going to list multiple filenames separated by spaces in place of {} (and it will be a long list, since you have 100000 files) rather than just a single filename. That being the case, the {} is required to come at the end of the command.

See the find(1) man page under -exec command {} +.

Answer 3

Assuming all directories and files have regular names, i.e. not containing spaces, newlines or similar, this should work even with a huge number of files:

find . -iname "*.extension" -exec sh -c '
command="<command>"
additionalParameters="<additional parameters>"
h=$(($#/2))
cmd="$command "
for i in $(seq 1 $h);do
        cmd="$cmd $(eval echo \$$i) "
done
cmd="$cmd $additionalParameters"
$cmd
shift $h
$command "$@" $additionalParameters' sh {} +

Rationale:

When using the + punctuation, find builds a command as large as possible. There are two limitations involved, the maximum number of arguments allowed (should be 128k on Gnu/Linux) and the maximum size of the argument list (should be 2 MB on Gnu/Linux). The issue is the command called requires extra arguments (additional parameters). Adding them overflows the limit leading to the "too many arguments error". The script I suggest split the built parameter list in two parts and run two commands instead of one per block so adding extra arguments do not exhibit the issue.

Answer 4

You can use this script:

#! /bin/bash

cmd=echo

test $# -gt 2 || exit 2
num_trailing_args="$1"
[[ $num_trailing_args =~ ^(0|[1-9][0-9]*)$ ]] ||
  { echo "Illegal first argument ('${num_trailing_args}'); aborting"; exit 2; }
test $# -lt $((num_trailing_args+2)) &&
  { echo "Too few arguments; aborting"; exit 2; }
shift
trailing_args=()
for((i=0;i<num_trailing_args;i++)); do
        trailing_args[i]="$1"
        shift
done

"$cmd" "$@" "${trailing_args[@]}"

and then use

find ... -exec args_change_script.sh 3 t1 t2 t3 {} +

The name of the command should not be longer than the name of the script (just to be sure).