33

I would like to run:

./a.out < x.dat > x.ans

for each *.dat file in the directory A.

Sure, it could be done by bash/python/whatsoever script, but I like to write sexy one-liner. All I could reach is (still without any stdout):

ls A/*.dat | xargs -I file -a file ./a.out

But -a in xargs doesn't understand replace-str 'file'.

Thank you for help.

1
  • In addition to the other good answers here, you can use the -a option of GNU xargs. Commented Dec 9, 2013 at 23:22

6 Answers 6

42

First of all, do not use ls output as a file list. Use shell expansion or find. See below for potential consequences of ls+xargs misuse and an example of proper xargs usage.

1. Simple way: for loop

If you want to process just the files under A/, then a simple for loop should be enough:

for file in A/*.dat; do ./a.out < "$file" > "${file%.dat}.ans"; done

2.pre1 Why not   ls | xargs ?

Here's an example of how bad things may turn if you use ls with xargs for the job. Consider a following scenario:

  • first, let's create some empty files:

    $ touch A/mypreciousfile.dat\ with\ junk\ at\ the\ end.dat
    $ touch A/mypreciousfile.dat
    $ touch A/mypreciousfile.dat.ans
    
  • see the files and that they contain nothing:

    $ ls -1 A/
    mypreciousfile.dat
    mypreciousfile.dat with junk at the end.dat
    mypreciousfile.dat.ans
    
    $ cat A/*
    
  • run a magic command using xargs:

    $ ls A/*.dat | xargs -I file sh -c "echo TRICKED > file.ans"
    
  • the result:

    $ cat A/mypreciousfile.dat
    TRICKED with junk at the end.dat.ans
    
    $ cat A/mypreciousfile.dat.ans
    TRICKED
    

So you've just managed to overwrite both mypreciousfile.dat and mypreciousfile.dat.ans. If there were any content in those files, it'd have been erased.


2. Using  xargs : the proper way with  find 

If you'd like to insist on using xargs, use -0 (null-terminated names) :

find A/ -name "*.dat" -type f -print0 | xargs -0 -I file sh -c './a.out < "file" > "file.ans"'

Notice two things:

  1. this way you'll create files with .dat.ans ending;
  2. this will break if some file name contains a quote sign (").

Both issues can be solved by different way of shell invocation:

find A/ -name "*.dat" -type f -print0 | xargs -0 -L 1 bash -c './a.out < "$0" > "${0%dat}ans"'

3. All done within find ... -exec

 find A/ -name "*.dat" -type f -exec sh -c './a.out < "{}" > "{}.ans"' \;

This, again, produces .dat.ans files and will break if file names contain ". To go about that, use bash and change the way it is invoked:

 find A/ -name "*.dat" -type f -exec bash -c './a.out < "$0" > "${0%dat}ans"' {} \;
5
  • 2
    +1 for mentionning not to parse the output of ls.
    – rahmu
    Commented Oct 7, 2011 at 9:31
  • 2
    Option 2 breaks when the filenames contains ".
    – thiton
    Commented Oct 7, 2011 at 11:30
  • 2
    Very good point, thanks! I'll update accordingly. Commented Oct 7, 2011 at 11:39
  • I want just mention, that if zsh is used as shell (and SH_WORD_SPLIT is not set), all the nasty special cases (white spaces, " in the filename etc.) need not to be considered. The trivial for file in A/*.dat; do ./a.out < $file > ${file%.dat}.ans ; done works in all cases.
    – jofel
    Commented Mar 20, 2012 at 19:56
  • 1
    -1 because I'm trying to figure out how to do xargs with stdin and my question has nothing to do with files or find.
    – user541686
    Commented May 7, 2019 at 0:04
3

Use GNU Parallel:

parallel ./a.out "<{} >{.}.ans" ::: A/*.dat

Added bonus: You get the processing done in parallel.

Watch the intro videos to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

4
  • 2
    You are the author parallel, so this falls under self-promotion without a disclaimer. Also, I don't like how Parallel lies to users about calling it with a flag to accept an agreement, but then prompts for additional input. Until you remove that, I doubt your software will receive widespread adoption. It's also annoying because I can't use it cleanly on a VM/container. Commented Oct 12, 2020 at 21:01
  • @EntangledLoops Thanks for your input. You can read the background on git.savannah.gnu.org/cgit/parallel.git/tree/doc/… (TL;DR: The goal is not widespread adoption, but long term survival).
    – Ole Tange
    Commented Oct 12, 2020 at 21:07
  • I see. I've tried your software and it worked fine, but I wasn't comfortable with the --citation (although I can certainly relate to your predicament). Despite your understandable desire for citations, it feels very non-GNU to force the command. My primary complaint was that it didn't work as described: it clearly says that you can run "parallel --citation" to never see the warning again, but that isn't true! You then need to manually enter something. I felt deceived and uninstalled it. Commented Oct 12, 2020 at 22:22
  • @EntangledLoops check how debian solved it, there was no long quarrel about it. after all parallel is open source, so you can remove the citation warning. It is indeed something I've not seen with any other GNU tool before. In general it's a well working tool, I prefer xargs for simplicity in most cases but sometimes parallel comes handy.
    – John
    Commented Dec 26, 2021 at 2:03
2

Try doing something like this (syntax may vary a bit depending on the shell you use):

$ for i in $(find A/ -name \*.dat); do ./a.out < ${i} > ${i%.dat}.ans; done

6
  • That would not work. It would try to operate on stuff like somefile.dat.dat and redirect all output to a single file. Commented Oct 7, 2011 at 8:54
  • You're right. I edited the solution to correct it.
    – rahmu
    Commented Oct 7, 2011 at 8:57
  • OK - Almost good :) Just somefile.dat.ans output stuff would look not so nice. Commented Oct 7, 2011 at 9:07
  • 1
    Edited! I did not know about '%'. It works like a charm, thanks for the tip.
    – rahmu
    Commented Oct 7, 2011 at 9:23
  • 1
    Adding a -type file would be nice (can't < directory), and this makes the unusual-filename-fairy sad.
    – Mat
    Commented Oct 7, 2011 at 9:57
2

For simple patterns, the for loop is appropriate:

for file in A/*.dat; do
    ./a.out < "${file}" > "${file%.dat}.ans" # Never forget the QUOTES!
done

For more complex cases where you need another utility to list the files (zsh or bash 4 have powerful enough patterns that you rarely need find, but if you want to stay within POSIX shell or use fast shell like dash, you will need find for anything non-trivial), while read is most appropriate:

find A -name '*.dat' -print | while IFS= read -r file; do
   ./a.out < "${file}" > "${file%.dat}.ans" # Never forget the QUOTES!
done

This will handle spaces, because read is (by default) line-oriented. It will not handle newlines and it will not handle backslashes, because by default it interprets escape sequences (that actually allows you to pass in a newline, but find can't generate that format). Many shells have -0 option to read so in those you can handle all characters, but unfortunately it's not POSIX.

1

I think you need at least a shell invocation in the xargs:

ls A/*.dat | xargs -I file sh -c "./a.out < file > file.ans"

Edit: It should be noted that this approach does not work when the filenames contain whitespace. Can't work. Even if you used find -0 and xargs -0 to make xargs understand the spaces correctly, the -c shell call would croak on them. However, the OP explicitely asked for an xargs solution, and this is the best xargs solution I came up with. If whitespace in filenames might be an issue, use find -exec or a shell loop.

5
  • Have a look at why it is a bad idea to parse ls output. Commented Oct 7, 2011 at 9:13
  • @rozcietrzewiacz: In general, sure, but I assume someone trying to do xargs voodoo knows this. Since these filenames undergo another shell expansion, they have to be well-behaved anyway.
    – thiton
    Commented Oct 7, 2011 at 9:19
  • 1
    You are wrong about the shell expansion. ls outputs things like spaces without escaping and that is the problem. Commented Oct 7, 2011 at 9:25
  • @rozcietrzewiacz: Yes, I understand that problem. Now just suppose you would properly escape these spaces and get them into xargs: They are replaced in sh's -c string, sh tokenizes the -c string, and everything breaks.
    – thiton
    Commented Oct 7, 2011 at 10:19
  • Yes. You see now, parsing ls is no good. But xargs can be safely used with find - see the suggestion No 2 in my answer. Commented Oct 7, 2011 at 10:53
1

There's no need to complicate it. You could do it with a for loop:

for file in A/*.dat; do
  ./a.out < "$file" >"${file%.dat}.ans"
done

the ${file%.dat}.ans bit will remove the .dat filename suffix from the filename in $file and instead add .ans to the end.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .