cut -d, -f2 parsed_repayment.csv | uniq -d | xargs -n1  -I{} grep {} parsed_repayment.csv | wc -l

I basically want to skip the first match for each grep command and write them to a file, but xargs combines all the output. Is there some handy way to achieve that ?

For example, say I have a list of words 'dog','cat', 'horse'. I want to search for all three words in a file using grep. Now suppose file contains 2 matching lines for each word-

dog l1
dog l2
cat l1
cat l2
horse l1
horse l2

The output I expect is -

dog l2
cat l2
horse l2

For each word searched, I want it to skip the first match. Also each word can have different number of matches.

I tried writing the output after grep to a temp file and then cut the first line using tail command, but as xargs is passing matches for all of the given words together, I am unable to skip first match for individual words.


4 Answers 4


It sounds like a XY problem and that you actually want:

awk -F, '$2 == prev; {prev = $2}' < parsed_repayment.csv

That is, report the lines where the second field is the same as the second field of the previous line.

For the answer to what you asked, as opposed to what you actually wanted, that would be:

cut -d, -f2 parsed_repayment.csv |
  uniq -d |
  xargs sh -c '
    for i do
      grep -e "$i" parsed_repayment.csv | tail -n +2
    done' sh

But that has a few caveats:

  • xargs expects a very specific input format. If the values of the second column contains blanks or single quotes or doubles quotes or backslashes, that won't work properly.
  • grep understands that $i as a regular expression. And even if you use -F or if those strings don't contain regular expression operators (like ., $...), grep would look for those strings anywhere in the line, not only in the second field, let alone exactly as the second field.
  • Quick question, how the for loop is accessing values passed by xargs, also what is the need to end the command with sh ?
    – Krrish Raj
    Commented Apr 30, 2018 at 11:45
  • @krrish, that sh is stored by sh inside $0. That is, it give a name (here sh, you could use any arbitrary string, but use something relevant as it shows up in error messages output by sh) to that inline-script. xargs passes the words on its input as additional arguments to sh, and that's what for i do loops over (it loops over the positional parameters). Commented Apr 30, 2018 at 11:51

You can use sed -e 1d as a pipe over grep inside xargs like this:

xargs -n1 -I{} bash -c 'grep {} apt-installed |sed -e 1d'

So finally:

cut -d, -f2 parsed_repayment.csv | uniq -d | xargs -n1 -I{} bash -c 'grep {} apt-installed |sed -e 1d' | wc -l


The following writes the first input line to file foo, and echoes the remainder:

read line ; echo "$line" >> foo ; while read line ; do echo "$line" ; done 

Test e.g. like this:

seq 0 9 | ( read line ; echo "$line" > foo ; while read line ; do echo "$line" ; done ) 
cat foo

Now instead of executing grep directly in xargs, execute a pipe which puts the output of grep through the above fragment. This solves the problem of xargs concatenating all output.

The whole thing is long enough that I'd consider putting the fragment into a shells script, esp. if you want to control the filename.


This assumes you want to write the entry you have skipped to a file, and the rest to stdout, but maybe I understood you wrongly.


With GNU Parallel it looks like this:

cut -d, -f2 parsed_repayment.csv | uniq -d |
  parallel 'grep {} parsed_repayment.csv | tail -n +2'

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .