Running a command multiple times with arguments (filenames) from a file?

Question

I have a file with a long list of filenames (with full paths). I've also got a program I'd like to run multiple times, using one and one filename from this list as argument. The program I'd like to run, is sadly a self-made script; so it can only take one filename at a time, and it cannot accept input from stdin (like a list of filenames) - so no piping or input-redirection possible.

What I need, is a command that will run another command (my script), using the lines in the file as argument.

Something like find with the -exec-action... (find . -name "*.txt" -exec command \; ). I guess I actually could use find here, but I'd like for the input-files to be sorted...

So what I need is something like this:

for_each_command1 -f list_of files -c './my_command {}'
for_each_command2 -f list_of_files -exec ./my_command {} \;
for_each_command3 -c './my_command {}' 

My usual way to handle such tasks is with sed - unfortunately, there's a lot of overhead and it's not pretty (but it does work...):


$ wc -l list_of_files
232
$ for (( i=1; i do
> ./my_command "`sed -n "${i}p" list_of_files`"
> done

So is there a command or shell built-in to handle something like this?

Answer 1

Conveniently, xargs -I does exactly what you want:

$ xargs <my_file_list.txt -I filename ./my_command "filename"

-I replace-str
    Replace occurrences of replace-str in the initial-arguments with names read
    from standard input.
    Also, unquoted blanks do not terminate input items;
    instead the separator is the newline character.
    Implies -x and -L 1.

This means it takes exactly one newline-delimited input line and invokes your command with it as a single argument. Nothing apart from newline is treated as a delimiter, so spaces and other special characters are fine.

Note that if you wanted to allow newlines in your filenames, the nul terminator (as per Wildcard's answer) will also work in a file.

Answer 2

I assumes your list of files is stored in a file named 'filelist.txt', each file name in a line. Then you can call your script with each line as an argument as follow:

while read name; do
  ./yoursrcipt.sh "$name"
done <filelist.txt

Answer 3

I usually prefer portable (POSIX) solutions, but the simple, obvious answer here is to use GNU extensions to make your life easier:

find . -type f -name '*.txt' -print0 | sort -z | xargs -n1 -r0 ./mycommand

This uses null bytes as separators between filenames, which is important since filenames can contain any character, including newlines.

(Slashes are also reserved, since they are path component separators, but that doesn't matter in this context because you can't use them as file delimiters.)

Answer 4

sed < list_of_files 's,^,./mycommand ",;s,$,",' | sh

Answer 5

$ cat baard.in
/path/to/some space file
/some   tabbed/some     tabbed  file
/path to/genericfile
/path/to/regular.file
/path/to/exciting!/file!
/path/to/(parenthetical)/file.txt

$ cat my_command
#!/bin/sh
printf "my_command: %s\n" "$1"

Since awk already parses lines, you could use it to call your command with each record, being careful to quote the parameter:

$ awk '{system("./my_command '\''" $0 "'\''");}' < baard.in
my_command: /path/to/some space file
my_command: /some       tabbed/some     tabbed  file
my_command: /path to/genericfile
my_command: /path/to/regular.file
my_command: /path/to/exciting!/file!
my_command: /path/to/(parenthetical)/file.txt

The awk command fails if there are single quotes in the input, though, such as:

/books/Hitchhiker's Guide to the Galaxy.epub

If you have bash available, you could use:

$ readarray -t paths < baard.in
$ for((i=0;i < ${#paths[*]}; i++)); do ./my_command "${paths[i]}"; done
my_command: /path/to/some space file
my_command: /some       tabbed/some     tabbed  file
my_command: /path to/genericfile
my_command: /path/to/regular.file
my_command: /path/to/exciting!/file!
my_command: /path/to/(parenthetical)/file.txt
my_command: /books/Hitchhiker's Guide to the Galaxy.epub

or in ksh93 (hat-tip to Gilles' answer here for the manual implementation of readarray):

$ IFS=$'\n'; set -f
$ paths=( $(< baard.in) )
$ unset IFS; set +f
$ for((i=0;i < ${#paths[*]}; i++)) do ./my_command "${paths[i]}"; done
my_command: /path/to/some space file
my_command: /some       tabbed/some     tabbed  file
my_command: /path to/genericfile
my_command: /path/to/regular.file
my_command: /path/to/exciting!/file!
my_command: /path/to/(parenthetical)/file.txt
my_command: /books/Hitchhiker's Guide to the Galaxy.epub

Answer 6

If I understand your question correctly, something like this should do what you need (assuming your file has one filename per line):

for i in `cat list_of_files`; do ./mycommand "$i"; done

Answer 7

as i understand it you have asked how to read an infile and run a task for each line read.

while IFS= read -r line
do    mycmd "$line"
done  < infile

or if you want to sort your input first:

sort ./infile[123] |
while IFS= read -r line
do    mycmd "$line"
done