1
cut -d, -f2 parsed_repayment.csv | uniq -d | xargs -n1  -I{} grep {} parsed_repayment.csv | wc -l

I basically want to skip the first match for each grep command and write them to a file, but xargs combines all the output. Is there some handy way to achieve that ?

For example, say I have a list of words 'dog','cat', 'horse'. I want to search for all three words in a file using grep. Now suppose file contains 2 matching lines for each word-

dog l1
dog l2
cat l1
cat l2
horse l1
horse l2

The output I expect is -

dog l2
cat l2
horse l2

For each word searched, I want it to skip the first match. Also each word can have different number of matches.

I tried writing the output after grep to a temp file and then cut the first line using tail command, but as xargs is passing matches for all of the given words together, I am unable to skip first match for individual words.

  • 1
    post a testable input fragment and expected result – RomanPerekhrest Apr 27 '18 at 11:29
  • So the first matched line should go to a file, and all others to stdout? Does the file have the same name for all input files? Also, you seem to take fields from a file and then grep for exactly these fields. Can you add some details on what you are trying to accomplish here, there might be easier ways to do this. – nohillside Apr 27 '18 at 11:30
  • would be great if you can provide some example, with what you are getting and what you are expecting to get – Sufiyan Ghori Apr 27 '18 at 11:37
  • if you want it to skip the first match then what wc -l for? – RomanPerekhrest Apr 27 '18 at 11:48
1

It sounds like a XY problem and that you actually want:

awk -F, '$2 == prev; {prev = $2}' < parsed_repayment.csv

That is, report the lines where the second field is the same as the second field of the previous line.

For the answer to what you asked, as opposed to what you actually wanted, that would be:

cut -d, -f2 parsed_repayment.csv |
  uniq -d |
  xargs sh -c '
    for i do
      grep -e "$i" parsed_repayment.csv | tail -n +2
    done' sh

But that has a few caveats:

  • xargs expects a very specific input format. If the values of the second column contains blanks or single quotes or doubles quotes or backslashes, that won't work properly.
  • grep understands that $i as a regular expression. And even if you use -F or if those strings don't contain regular expression operators (like ., $...), grep would look for those strings anywhere in the line, not only in the second field, let alone exactly as the second field.
  • Quick question, how the for loop is accessing values passed by xargs, also what is the need to end the command with sh ? – Krrish Raj Apr 30 '18 at 11:45
  • @krrish, that sh is stored by sh inside $0. That is, it give a name (here sh, you could use any arbitrary string, but use something relevant as it shows up in error messages output by sh) to that inline-script. xargs passes the words on its input as additional arguments to sh, and that's what for i do loops over (it loops over the positional parameters). – Stéphane Chazelas Apr 30 '18 at 11:51
0

You can use sed -e 1d as a pipe over grep inside xargs like this:

xargs -n1 -I{} bash -c 'grep {} apt-installed |sed -e 1d'

So finally:

cut -d, -f2 parsed_repayment.csv | uniq -d | xargs -n1 -I{} bash -c 'grep {} apt-installed |sed -e 1d' | wc -l

0

The following writes the first input line to file foo, and echoes the remainder:

read line ; echo "$line" >> foo ; while read line ; do echo "$line" ; done 

Test e.g. like this:

seq 0 9 | ( read line ; echo "$line" > foo ; while read line ; do echo "$line" ; done ) 
cat foo

Now instead of executing grep directly in xargs, execute a pipe which puts the output of grep through the above fragment. This solves the problem of xargs concatenating all output.

The whole thing is long enough that I'd consider putting the fragment into a shells script, esp. if you want to control the filename.

Edit

This assumes you want to write the entry you have skipped to a file, and the rest to stdout, but maybe I understood you wrongly.

0

With GNU Parallel it looks like this:

cut -d, -f2 parsed_repayment.csv | uniq -d |
  parallel 'grep {} parsed_repayment.csv | tail -n +2'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.