275

I want to run a java command once for every match of ls | grep pattern -. In this case, I think I could do find pattern -exec java MyProg '{}' \; but I'm curious about the general case - is there an easy way to say "run a command once for every line of standard input"? (In fish or bash.)

0

13 Answers 13

318

The accepted answer has the right idea, but the key is to pass xargs the -n1 switch, which means "Use at most 1 argument per command line":

cat file... | xargs -n1 command

Or, for a single input file you can avoid the pipe from cat entirely and just go with:

<file xargs -n1 command

Updated 2020-08-05:

I would also like to respond to the advice found in user Jander's comment, which was heavily upvoted, despite containing some amount of misinformation as I will now explain.

Do not be so hasty to recommend the -L option of xargs, without mentioning the trouble that its (so called) trailing blank(s) feature can lead to. In my opinion this switch causes more harm than good and is certainly a stretch to use to mean, for the case of -L 1, act on one non-empty line at a time. To be fair, the man page for xargs does spell out the features (i.e., issues) that come for the ride with the -L switch.

Since Jander made no mention of the issues when mentioning -L to perhaps a hasty unsuspecting StackOverflow audience seeking quick tips and not having the time for such tedious things as reading man pages rather than accepting comments and answers as gospel, I will now present my case for why -L is a very bad suggestion without a careful understanding of all of the baggage that it brings along for the ride.

To illustrate my disdain for -L, let's consider a simple input file that consists of the following text someone carelessly entered (perhaps a high-school summer intern that created this data file as part of his/her training, as evidenced by its Windowish filename. As luck (karma?) would have it, you have been selected by management to be its new custodian):

testdata.txt

1
2␠
3

Because of the fact that the line that contains the digit 2 has a space character (shown as a Unicode SYMBOL FOR SPACE glyph after the digit 2 in the preceding code, in case your browser's font does not have a visual representation for this character), a command that uses xargs -L1, such as:

<testdata.txt xargs -L1 echo

..., would produce the following (perhaps surprising) output:

1
2 3

This is caused by the fact that the -L switch instructs xargs to append subsequent lines to those that end with blanks, a behavior that may only effect the resulting output in those oddball moments where lines are not properly trimmed of trailing blanks - a time bomb bug waiting for the right input file to present itself.

On the other hand, the same command using the -n 1 switch of xargs, instead of -L 1 would produce a far more acceptable output of:

1
2␠
3

And that's not even the worst of it! The -L switch unlike -n forces the "dreaded" -x option of xargs to go into effect. This causes termination of the xargs process if a command line is encountered that it deems too long for the environment on which it is run.

An input file consisting of many lines with trailing blanks in succession could, as instructed by the -L switch and its use of the chemical agent known as Agent -x into the mix, potentially cause xargs to terminate midstream if the concatenation of all of these into one superline exceeds xargs' definition of line is too long for a command line. If things are beginning to look murky, consider that line is too long is a size determined by xargs based on the max length specified for the platform on which it is run and further offset by a seemingly arbitrary constant as explained in more detail in the man page. Remember those pesky indefinite integrals from Calculus and their arbitrary constants and losing a point on a quiz or test, because you forgot to write + C after your solution for an indefinite integral? Well, that phrase is back with a vengeance, to bite you on the rear once again, if add -L to your handy xargs toolkit.

A -n value of 1, on the other hand, would just chop up those long lines into (hopefully) small bite sized one-line chunks and execute the command supplied to xargs for each of them, one at a time, without any consideration given to whether they end with blanks or not. No more long lines and no more of xargs stabbing you in the back by terminating abruptly - Et tu, Brute -x?

An optional segue regarding phrasing in the xargs man page

I don't know why the ambiguous and non-standard word blanks was used throughout the xargs man page, instead of far better defined and less ambiguous options such as:

  • space(s), if blanks means one or more ASCII space characters
  • whitespace(s) other than new-lines (if that's what blanks implies)
  • one or more non-printable characters from the set: {space, horizontal tab} (if blanks was used as a synonym for this gruesome twosome)

Updated 2021-06-15:

User @BjornW asked how xargs could be used to run a command once per line of input and not just word of input. (See, I do read the comments, and I'll just blame the seven months it took to respond on Covid :P ).

In the spirit of the original question, as asked, and to make my answer applicable to a greater number of use cases, I would like to address this particular scenario in detail.

Consider the following input file. It is chock-full of various edge cases one may actually encounter in the Real World™ (e.g., leading/trailing spaces, lines consisting only of spaces, empty lines, lines beginning with a hyphen [which should not get interpreted as the introduction of a switch], etc.):

lines.txt

a1 a22 a333 a4444
b4444 b333 b22 b1
␠␠c d e f g
␣
hhh
ii jj kk␠
␣
␠␠␠
-L and -x are the gruesome twosome
␣
␣
␣

In the preceding input file, the Unicode character OPEN BOX U+2423 was used to mark empty lines and the Unicode SYMBOL FOR SPACE was used for leading and trailing spaces, in order to make them more prominent.

Let's say we want to run a command on each line of input, taken as a whole, and passed to our command as a single argument, regardless of content (including no content). We would go about this using xargs, as follows (Note: printf will be our sample command and the %q format specifier will be used in order to enclose the supplied argument in apostrophes for clarity, when spaces are present or the argument is an empty string - all in, only our hhh input line was left "unscathed," by %q, as you will see in the output, presented shortly. Had any non-printable characters been present, they would have also gotten escaped by %q using the POSIX $'' quoting syntax]):

<lines.txt xargs -n1 -d'\n' printf -- 'Input line: %q\n'

The output is as follows:

Input line: 'a1 a22 a333 a4444'
Input line: 'b4444 b333 b22 b1'
Input line: '   c d e f g'
Input line: ''
Input line: hhh
Input line: 'ii jj kk '
Input line: ''
Input line: '   '
Input line: '-L and -x are the gruesome twosome'
Input line: ''
Input line: ''
Input line: ''

So, there you have it. Using the -d switch, we can specify the delimiter that xargs should look for in our input file to indicate where an argument ends and the next one begins. By setting it to '\n' which xargs itself is smart enough to interpret as a C-style character escape, as stated in the description of the -d switch on its man page, we can use xargs to forward entire lines of input to our command of choice as arguments, with minimal effort on our part.

I would like to also mention that xargs can be used to concatenate multiple lines of input (with a caveat I will detail at the end of this paragraph), for the rare cases when such behavior is desired, and forward them as a single argument to our command. This can be accomplished by setting the number passed to the -n switch to xargs in the above invocation command, to a value that is indicative of the number of lines of input that should get merged into a single argument with their \n line endings removed as part of the process. Unfortunately, this new-line stripping behavior makes the aforementioned xargs approach unsuitable for many use cases, because information that is indicative of where one line ends and the next begins, gets lost in the process.

11
  • 2
    Also of interest is the ability of xargs to not run if stdin is empty: --no-run-if-empty -r : If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if there is no input. This option is a GNU extension. Oct 24, 2015 at 19:58
  • 11
    How dyou access the line inside command ?
    – B T
    Apr 14, 2016 at 1:21
  • This is the correct use of xargs. Without -n1, it only works on commands that treat lists of parameters as multiple invocations which not all do.
    – masterxilo
    Mar 15, 2018 at 9:43
  • 4
    printf "foo bar\nbaz bat" | xargs -n1 echo whee splits by words and not by lines Oct 24, 2018 at 8:32
  • @B T, I think you have a fundamental misunderstanding of the role that xargs plays. Consider (and try for yourself) the command: yes | head -n 1 | xargs -n 1 echo. It is logically the same as just saying: echo 'y' . Where did that y come from, you ask? It is the first line of yes' output. What you are referring to as accessing the line is the one and only argument that xargs passes to the command when the -n 1 switch is used with it. In other words, for each line of input, command gets executed with that line passed in as an argument. Aug 5, 2020 at 4:13
148

In Bash or any other Bourne-style shell (ash, ksh, zsh, …):

while read -r line; do command "$line"; done

read -r reads a single line from standard input (read without -r interprets backslashes, you don't want that). Thus you can do either of the following:

$ command | while read -r line; do command "$line"; done  

$ while read -r line; do command "$line"; done <file
3
  • 9
    When I tried tail -f syslog | grep -e something -e somethingelse| while read line; do echo $line; done it didn't work. It worked with a file piped into the while loop, worked with just the tail -f, worked with just grep, but not with both pipes. Giving the grep the --line-buffered option made it work
    – user16468
    Mar 10, 2012 at 20:39
  • 1
    This works also when each line needs to be sent to stdin: command | while read -r line; do echo "$line" | command ; done
    – Den
    Sep 10, 2018 at 9:27
  • Beware of the intricacies of read though! (E.g. it trims spaces, and may do other unexpected things!)
    – anon
    Aug 16, 2023 at 10:47
126

That's what xargs does.

... | xargs command
7
  • 41
    Not quite. printf "foo bar\nbaz bat" | xargs echo whee will yield whee foo bar baz bat. Maybe add the -L or -n options?
    – Jander
    Feb 17, 2011 at 5:17
  • 5
    @Jander The question was rather general, so I gave the general tool. True, you will have to adjust its behavior with options depending on the specific circumstances.
    – Keith
    Feb 17, 2011 at 5:25
  • 5
    ... | tr '\n' '\0' | xargs -0
    – vrdhn
    Feb 17, 2011 at 11:12
  • 15
    like, "the specific circumstances which give the right answer to the question". :)
    – mattdm
    Feb 17, 2011 at 18:17
  • 15
    If you want to see the proper way to do this with xargs, see my answer below. Jun 30, 2015 at 16:08
26

I agree with Keith, xargs is the most general tool for the job.

I usually use a 3 step approach.

  • do the basic stuff until you have something you would like to work with
  • prepare the line with awk so it gets the correct syntax
  • then let xargs execute it, maybe with the help of bash.

There are smaller and faster ways, but this ways almost always works.

A simple example:

ls | 
grep xls | 
awk '{print "MyJavaProg --arg1 42 --arg2 "$1"\0"}' | 
xargs -0 bash -c

the 2 first lines selects some files to work with, then awk prepares a nice string with a command to execute and some arguments and $1 is the first column input from the pipe. And finally I make sure that xargs sends this string to bash that just execute it.

It is a little bit overkill, but this recipe has helped me in a lot of places since it is very flexible.

Also note, you can also do xargs -0 -n1 bash -c (just adding the -n1 flag suggested by Michael Goldshteyn) to execute the command on each line of output.

3
  • 7
    Note, xargs -0 uses the null byte as a record separator, so your awk print statement should be printf("MyJavaProg --args \"%s\"\0",$1) Feb 17, 2011 at 14:30
  • @glenn: Missed the null char, will update the answer
    – Johan
    Feb 18, 2011 at 12:43
  • @Johan not a big deal, but if you're using awk you can have it do the pattern match and skip the grep e.g., ls | awk '/xls/ {print... May 31, 2016 at 15:19
18

GNU Parallel is made for that kind of tasks. The simplest usage is:

cat stuff | grep pattern | parallel java MyProg

Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

3
  • 2
    No real need for the cat here since grep can directly read the file May 31, 2016 at 15:20
  • 3
    @EricRenouf oletange.blogspot.dk/2013/10/useless-use-of-cat.html
    – Ole Tange
    May 31, 2016 at 20:29
  • 1
    Thanks for the link, I don't necessarily agree that it's easier to read, but nice to know it was considered regardless. I would only now slightly quibble that the link doesn't actually apply here since the alternative is not really < stuff grep pattern but is grep pattern stuff with no redirection or cat required at all. Still, that doesn't materially change your argument and if you think it's clearer to always use things in a pipe that starts with cat, then power to you Jun 1, 2016 at 0:32
17

If you need to control where exactly the input argument is inserted into your command line or if you need to repeat it several times then you can use xargs -I{}.

EXAMPLE #1

Create an empty folder structure in another_folder that mirrors the subfolders in the current directory:

    ls -1d ./*/ | xargs -I{} mkdir another_folder/{}
EXAMPLE #2

Apply an operation on a file list coming from stdin, in this case make a copy of each .html file by appending a .bak extension:

    find . -iname "*.html" | xargs -I{} cp {} {}.bak

From the xargs man page for MacOS/BSD:

 -I replstr
         Execute utility for each input line, replacing one or more occurrences of
         replstr in up to replacements (or 5 if no -R flag is specified) arguments
         to utility with the entire line of input.  The resulting arguments, after
         replacement is done, will not be allowed to grow beyond 255 bytes; this is
         implemented by concatenating as much of the argument containing replstr as
         possible, to the constructed arguments to utility, up to 255 bytes.  The
         255 byte limit does not apply to arguments to utility which do not contain
         replstr, and furthermore, no replacement will be done on utility itself.
         Implies -x.

Linux xargs man page:

   -I replace-str
          Replace  occurrences of replace-str in the initial-
          arguments with names read from standard input.  Al‐
          so,  unquoted  blanks do not terminate input items;
          instead the separator  is  the  newline  character.
          Implies -x and -L 1.
12

Also, while read loop in fish shell (I assume you want fish shell, considering you used tag).

command | while read line
    command $line
end

Few points to note.

  • read doesn't take -r argument, and it doesn't interpret your backslashes, in order to make most common use case easy.
  • You don't need to quote $line, as unlike bash, fish doesn't separate variables by spaces.
  • command by itself is a syntax error (to catch such use of placeholder arguments). Replace it with the real command.
3
  • 1
    Doesn't while need to be paired with do & done instead of end?
    – aff
    Nov 19, 2018 at 7:55
  • 1
    @aff This is specifically about fish shell, which has different syntax.
    – 0..
    Nov 19, 2018 at 14:29
  • Ah, so that's what the fish means.
    – aff
    Nov 20, 2018 at 6:33
4

Here, a copypaste you can immediately use:

cat list.txt | xargs -I{} command parameter {} parameter

The item from the list will be put where the {} is and the rest of the command and parameters will be used as-is.

3

This is possible with xargs, as other answers indicated. We need to distinguish two specifics in the "once per line" part of the question:

  1. Once: Use -n 1, this ensures that the command is invoked exactly once for each argument. However, by default, xargs assumes that arguments are space-delimited -- this will break once the files contain spaces.
  2. Per line: Use -d '\n', or preprocess the input with tr '\n' '\0' and use -0 . This makes the command robust against spaces in the input.

The final command line then becomes:

.... | xargs -n 1 -d '\n' <command>

or (with tr)

.... | tr '\n' '\0' | xargs -n 1 -0 <command>

If your command can process multiple arguments at once (like grep or sed), you can omit -n 1 to speed up the operation in many cases.

1

When dealing with potentially unsanitized inputs, I like to see the entire job 'spelled out' line by line for visual inspection before I run it (especially when it's something destructive like cleaning people's mailbox's).

So what I do is generate a list of parameters (ie. usernames), feed it to a file in one-record-per-line fashion, like this:

johndoe  
jamessmith  
janebrown  

Then I open the list in vim, and mangle it with search and replace expressions until I get a list of full commands that need to get executed, like this:

/bin/rm -fr /home/johndoe  
/bin/rm -fr /home/jamessmith 

This way if your regex is incomplete, you will see in what command will have potential problems (ie. /bin/rm -fr johnnyo connor). This way you can undo your regex, and try it again with a more reliable version of it. Name mangling is notorious for this, because it's hard to take care of all the edge cases like Van Gogh, O'Connors, St. Clair, Smith-Wesson.

Having set hlsearch is useful for doing this in vim, as it will highlight all the matches, so you can easily spot if it doesn't match, or matches in an unintended way.

Once your regex is perfect and it catches all the cases you can test for/think of, then I usually convert it to a sed expression so it can be fully automated for another run.

For cases where the number of lines of input prevents you from doing a visual inspection, I highly recommend echoing the command to the screen (or better yet, a log) before it executes, so if it errors out, you know exactly which command caused it to fail. Then you can go back to your original regex and adjust once more.

0

If a program ignores the pipe but accepts files as arguments, then you can just point it to the special file /dev/stdin.

I am not familiar with java, but here is an example of how you would do it for bash:

$ echo $'pwd \n cd / \n pwd' |bash /dev/stdin
/home/rolf
/

The $ is necessary for bash to translate \n into newlines. I'm not sure why.

0

I prefer this - allowing multi-line commands and clear code

find -type f -name filenam-pattern* | while read -r F
do
  echo $F
  cat $F | grep 'some text'
done

ref https://stackoverflow.com/a/3891678/248616

0

This should work for everything,

  • including self-defined functions,
  • without spawning any additional processes, and
  • without removing spaces (which read otherwise does!).

Note the IFS= and -r, not included in any other answer:

mapp() { while IFS= read -r line; do "$1" "$line"; done; }

Here’s an example usage:

$ bla() { echo "  bla: $1"; }
$ echo -e "1\n2\n3\n4" | mapp bla 
  bla: 1
  bla: 2
  bla: 3
  bla: 4

Here’s an alternative implementation that allows passing multiple arguments instead:

mapa() { f="$1"; shift; for x in "$@"; do "$f" "$x"; done; }

Example:

$ bla() { echo "  bla: $1"; }
$ mapa bla 1 2 3 4
  bla: 1
  bla: 2
  bla: 3
  bla: 4

Finally, some universal ones:

# Doesn’t care where the input comes from. (First parameters, then pipe lines.)
map() { f="$1"; shift
  if [[ $# -gt 0 ]]; then for x in "$@"; do "$f" "$x"; done; fi
  if [ ! -t 0 ] ; then while IFS= read -r line; do "$f" "$line"; done; fi
}
# Cross product
# (The parameters are the things to call,
#  the pipe takes values to call those things with.)
cross() {
  while IFS= read -r x; do
    for f in "$@"; do
      "$f" "$x"
    done
  done
}  

Examples:

$ bla() { echo "bla: $1"; }
$ blubb() { echo "blubb: $1"; }
$ gnah() { echo "gnah: $1"; }

$ echo -e "x\ny\nz" | map bla 1 2 3
bla: 1
bla: 2
bla: 3
bla: x
bla: y
bla: z

$ echo -e "1\n2\n3\n4" | cross bla blubb gnah
bla: 1
blubb: 1
gnah: 1
bla: 2
blubb: 2
gnah: 2
bla: 3
blubb: 3
gnah: 3
bla: 4
blubb: 4
gnah: 4

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .