3

I have a file with a long list of filenames (with full paths). I've also got a program I'd like to run multiple times, using one and one filename from this list as argument. The program I'd like to run, is sadly a self-made script; so it can only take one filename at a time, and it cannot accept input from stdin (like a list of filenames) - so no piping or input-redirection possible.

What I need, is a command that will run another command (my script), using the lines in the file as argument.

Something like find with the -exec-action... (find . -name "*.txt" -exec command \; ). I guess I actually could use find here, but I'd like for the input-files to be sorted...

So what I need is something like this:

for_each_command1 -f list_of files -c './my_command {}'
for_each_command2 -f list_of_files -exec ./my_command {} \;
for_each_command3 -c './my_command {}' 

My usual way to handle such tasks is with sed - unfortunately, there's a lot of overhead and it's not pretty (but it does work...):

$ wc -l list_of_files 232 $ for (( i=1; i do > ./my_command "`sed -n "${i}p" list_of_files`" > done

So is there a command or shell built-in to handle something like this?

  • 1
    Also see Why is looping over find's output bad practice? which covers how to correctly use find ... -exec. – Wildcard Jan 10 '17 at 2:18
  • @Wildcard Usually it's not - and I'd usually do just that... But this time I wanted the file-list - and the subsequent list resulting from my script - to be in alphabetical order, and find does not output files in alphabetical order. Yes, you can sort the output of find ... -print(which is what I've done to make the file-list), but the file-order from find is not alphabetical. What my script does to the list - the information it parse from the files in the list - makes it far easier to sort the file-list first, than try to alphabetically sort the result afterwards. – Baard Kopperud Jan 10 '17 at 8:22
  • 1
    That doesn't actually make a difference; there are still better solutions. File globbing (including zsh file globbing extensions) are one approach. I've posted another, even simpler answer below. I still recommend you actually read through all the information at the page I linked above. – Wildcard Jan 10 '17 at 9:23
4

Conveniently, xargs -I does exactly what you want:

$ xargs <my_file_list.txt -I filename ./my_command "filename"
-I replace-str
    Replace occurrences of replace-str in the initial-arguments with names read
    from standard input.
    Also, unquoted blanks do not terminate input items;
    instead the separator is the newline character.
    Implies -x and -L 1.

This means it takes exactly one newline-delimited input line and invokes your command with it as a single argument. Nothing apart from newline is treated as a delimiter, so spaces and other special characters are fine.

Note that if you wanted to allow newlines in your filenames, the nul terminator (as per Wildcard's answer) will also work in a file.

  • Apart from being slightly obfuscated by having the redirection first on the command line, this is a good solution. – Kusalananda Jan 10 '17 at 11:29
  • Moved - I tend to put input redirection at the start, and output redirection at the end, out of some probably-misplaced feeling of symmetry. – Useless Jan 10 '17 at 11:52
3

I usually prefer portable (POSIX) solutions, but the simple, obvious answer here is to use GNU extensions to make your life easier:

find . -type f -name '*.txt' -print0 | sort -z | xargs -n1 -r0 ./mycommand

This uses null bytes as separators between filenames, which is important since filenames can contain any character, including newlines.

(Slashes are also reserved, since they are path component separators, but that doesn't matter in this context because you can't use them as file delimiters.)

2

I assumes your list of files is stored in a file named 'filelist.txt', each file name in a line. Then you can call your script with each line as an argument as follow:

while read name; do
  ./yoursrcipt.sh "$name"
done <filelist.txt
  • Assuming the file names have e.g. no spaces in them. Better to set the $IFS for the read. – phk Jan 9 '17 at 23:25
  • @phk Oh there are spaces - lots of 'em... and exclamation-marks, and parenthesis... – Baard Kopperud Jan 10 '17 at 0:48
  • @BaardKopperud Then use while IFS= read -r name; do. – Kusalananda Jan 10 '17 at 11:27
1
sed < list_of_files 's,^,./mycommand ",;s,$,",' | sh
1
$ cat baard.in
/path/to/some space file
/some   tabbed/some     tabbed  file
/path to/genericfile
/path/to/regular.file
/path/to/exciting!/file!
/path/to/(parenthetical)/file.txt

$ cat my_command
#!/bin/sh
printf "my_command: %s\n" "$1"

Since awk already parses lines, you could use it to call your command with each record, being careful to quote the parameter:

$ awk '{system("./my_command '\''" $0 "'\''");}' < baard.in
my_command: /path/to/some space file
my_command: /some       tabbed/some     tabbed  file
my_command: /path to/genericfile
my_command: /path/to/regular.file
my_command: /path/to/exciting!/file!
my_command: /path/to/(parenthetical)/file.txt

The awk command fails if there are single quotes in the input, though, such as:

/books/Hitchhiker's Guide to the Galaxy.epub

If you have bash available, you could use:

$ readarray -t paths < baard.in
$ for((i=0;i < ${#paths[*]}; i++)); do ./my_command "${paths[i]}"; done
my_command: /path/to/some space file
my_command: /some       tabbed/some     tabbed  file
my_command: /path to/genericfile
my_command: /path/to/regular.file
my_command: /path/to/exciting!/file!
my_command: /path/to/(parenthetical)/file.txt
my_command: /books/Hitchhiker's Guide to the Galaxy.epub

or in ksh93 (hat-tip to Gilles' answer here for the manual implementation of readarray):

$ IFS=$'\n'; set -f
$ paths=( $(< baard.in) )
$ unset IFS; set +f
$ for((i=0;i < ${#paths[*]}; i++)) do ./my_command "${paths[i]}"; done
my_command: /path/to/some space file
my_command: /some       tabbed/some     tabbed  file
my_command: /path to/genericfile
my_command: /path/to/regular.file
my_command: /path/to/exciting!/file!
my_command: /path/to/(parenthetical)/file.txt
my_command: /books/Hitchhiker's Guide to the Galaxy.epub
0

If I understand your question correctly, something like this should do what you need (assuming your file has one filename per line):

for i in `cat list_of_files`; do ./mycommand "$i"; done
  • It ought to... but I've found that it often doesn't work with filenames with spaces and such embedded (each word is treated as a filename) - hence my sed-hack. It's possible there are some options that prevent this, but in general I've had little success with cating and xargsing long lists of files. – Baard Kopperud Jan 10 '17 at 0:45
  • for performs usual expansion and splitting on whatever comes after in. There's no reliable way to do this with cat. – Useless Jan 10 '17 at 12:00
  • @BaardKopperud: the use of " around $i should take care of spaces and other unusual characters in the filename (except for *, ? and other special characters which get expanded by the shell). – Alessandro Dotti Contra Jan 10 '17 at 14:29
0

as i understand it you have asked how to read an infile and run a task for each line read.

while IFS= read -r line
do    mycmd "$line"
done  < infile

or if you want to sort your input first:

sort ./infile[123] |
while IFS= read -r line
do    mycmd "$line"
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.