Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I would like to run:

./a.out < x.dat > x.ans

for each *.dat file in the directory A.

Sure, it could be done by bash/python/whatsoever script, but I like to write sexy one-liner. All I could reach is (still without any stdout):

ls A/*.dat | xargs -I file -a file ./a.out

But -a in xargs doesn't understand replace-str 'file'.

Thank you for help.

share|improve this question
    
In addition to the other good answers here, you can use the -a option of GNU xargs. –  James Youngman Dec 9 '13 at 23:22
add comment

6 Answers

up vote 15 down vote accepted

First of all, do not use ls output as a file list. Use shell expansion or find. See below for potential consequences of ls+xargs misuse and an example of proper xargs usage.

1. Simple way: for loop

If you want to process just the files under A/, then a simple for loop should be enough:

for file in A/*.dat; do ./a.out < "$file" > "${file%.dat}.ans"; done

2.pre1 Why not   ls | xargs ?

Here's an example of how bad things may turn if you use ls with xargs for the job. Consider a following scenario:

  • first, let's create some empty files:

    $ touch A/mypreciousfile.dat\ with\ junk\ at\ the\ end.dat
    $ touch A/mypreciousfile.dat
    $ touch A/mypreciousfile.dat.ans
    
  • see the files and that they contain nothing:

    $ ls -1 A/
    mypreciousfile.dat
    mypreciousfile.dat with junk at the end.dat
    mypreciousfile.dat.ans
    
    $ cat A/*
    
  • run a magic command using xargs:

    $ ls A/*.dat | xargs -I file sh -c "echo TRICKED > file.ans"
    
  • the result:

    $ cat A/mypreciousfile.dat
    TRICKED with junk at the end.dat.ans
    
    $ cat A/mypreciousfile.dat.ans
    TRICKED
    

So you've just managed to overwrite both mypreciousfile.dat and mypreciousfile.dat.ans. If there were any content in those files, it'd have been erased.


2. Using  xargs : the proper way with  find 

If you'd like to insist on using xargs, use -0 (null-terminated names) :

find A/ -name "*.dat" -type f -print0 | xargs -0 -I file sh -c './a.out < "file" > "file.ans"'

Notice two things: 1: this way you'll create files with .dat.ans ending; 2: this will break if some file name contains a quote sign ("). Both issues can be solved by different way of shell invocation:

find A/ -name "*.dat" -type f -print0 | xargs -0 -L 1 bash -c './a.out < "$0" > "${0%dat}ans"'

3. All done within find ... -exec

 find A/ -name "*.dat" -type f -exec sh -c './a.out < "{}" > "{}.ans"' \;

This, again, produces .dat.ans files and will break if file names contain ". To go about that, use bash and change the way it is invoked:

 find A/ -name "*.dat" -type f -exec bash -c './a.out < "$0" > "${0%dat}ans'" {} \;
share|improve this answer
    
+1 for mentionning not to parse the output of ls. –  rahmu Oct 7 '11 at 9:31
1  
Option 2 breaks when the filenames contains ". –  thiton Oct 7 '11 at 11:30
1  
Very good point, thanks! I'll update accordingly. –  rozcietrzewiacz Oct 7 '11 at 11:39
    
I want just mention, that if zsh is used as shell (and SH_WORD_SPLIT is not set), all the nasty special cases (white spaces, " in the filename etc.) need not to be considered. The trivial for file in A/*.dat; do ./a.out < $file > ${file%.dat}.ans ; done works in all cases. –  jofel Mar 20 '12 at 19:56
add comment

Use GNU Parallel:

parallel ./a.out "<{} >{.}.ans" ::: A/*.dat

Added bonus: You get the processing done in parallel.

Watch the intro videos to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

share|improve this answer
add comment

For simple patterns, the for loop is appropriate:

for file in A/*.dat; do
    ./a.out < "${file}" > "${file%.dat}.ans" # Never forget the QUOTES!
done

For more complex cases where you need another utility to list the files (zsh or bash 4 have powerful enough patterns that you rarely need find, but if you want to stay within POSIX shell or use fast shell like dash, you will need find for anything non-trivial), while read is most appropriate:

find A -name '*.dat' -print | while IFS= read -r file; do
   ./a.out < "${file}" > "${file%.dat}.ans" # Never forget the QUOTES!
done

This will handle spaces, because read is (by default) line-oriented. It will not handle newlines and it will not handle backslashes, because by default it interprets escape sequences (that actually allows you to pass in a newline, but find can't generate that format). Many shells have -0 option to read so in those you can handle all characters, but unfortunately it's not POSIX.

share|improve this answer
add comment

I think you need at least a shell invocation in the xargs:

ls A/*.dat | xargs -I file sh -c "./a.out < file > file.ans"

Edit: It should be noted that this approach does not work when the filenames contain whitespace. Can't work. Even if you used find -0 and xargs -0 to make xargs understand the spaces correctly, the -c shell call would croak on them. However, the OP explicitely asked for an xargs solution, and this is the best xargs solution I came up with. If whitespace in filenames might be an issue, use find -exec or a shell loop.

share|improve this answer
    
Have a look at why it is a bad idea to parse ls output. –  rozcietrzewiacz Oct 7 '11 at 9:13
    
@rozcietrzewiacz: In general, sure, but I assume someone trying to do xargs voodoo knows this. Since these filenames undergo another shell expansion, they have to be well-behaved anyway. –  thiton Oct 7 '11 at 9:19
1  
You are wrong about the shell expansion. ls outputs things like spaces without escaping and that is the problem. –  rozcietrzewiacz Oct 7 '11 at 9:25
    
@rozcietrzewiacz: Yes, I understand that problem. Now just suppose you would properly escape these spaces and get them into xargs: They are replaced in sh's -c string, sh tokenizes the -c string, and everything breaks. –  thiton Oct 7 '11 at 10:19
    
Yes. You see now, parsing ls is no good. But xargs can be safely used with find - see the suggestion No 2 in my answer. –  rozcietrzewiacz Oct 7 '11 at 10:53
add comment

Try doing something like this (syntax may vary a bit depending on the shell you use):

$ for i in $(find A/ -name \*.dat); do ./a.out < ${i} > ${i%.dat}.ans; done

share|improve this answer
    
That would not work. It would try to operate on stuff like somefile.dat.dat and redirect all output to a single file. –  rozcietrzewiacz Oct 7 '11 at 8:54
    
You're right. I edited the solution to correct it. –  rahmu Oct 7 '11 at 8:57
    
OK - Almost good :) Just somefile.dat.ans output stuff would look not so nice. –  rozcietrzewiacz Oct 7 '11 at 9:07
1  
Edited! I did not know about '%'. It works like a charm, thanks for the tip. –  rahmu Oct 7 '11 at 9:23
1  
Adding a -type file would be nice (can't < directory), and this makes the unusual-filename-fairy sad. –  Mat Oct 7 '11 at 9:57
show 1 more comment

You could do it with a for loop:

for file in A/*.dat; do
  ./a.out < ${file} >${file%.dat}.ans
done
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.