0

I'm trying to find out how to filter specific csv files by name and then extract their second row into a single csv file. I'm not sure whether it is a good idea to find files and then pipe it with commands like sed/awk to extract desired line.

find -name "data.csv" | sed -n 2p > final.csv 

update

Using a for loop in bash script generates desired output but prints all output in a single line.

for OUTPUT in $(find -name "data.csv")
do

        sed -n 2p $OUTPUT

done

Please help me with your suggestions.

  • Why are you appending "\n"? The shell sees that as "n", and passes it on to sed, which interprets that as the name of an input file. And then, as you see, it complains that there's no file or directory called "n". – arensb Feb 1 '16 at 17:01
  • Yeah, it was silly try! – encodeflush Feb 1 '16 at 17:08
1

If you look at the output of find -name "data.csv", you'll see that it prints the full pathnames of files named data.csv. The pipeline passes that on to sed, which prints the second line of its input.

So what you're saying is, "Here's a list of files. Give me the second one one the list." What you really want to say is "Here's a list of files. For each one, give me the second line." And for that, you want to use xargs.

Most likely, you'll need

find -name "data.csv" -print0 | xargs -0 -n 1 sed -n 2p > final.csv

It's often a good idea to use find -print0 | xargs -0: the -print0 causes find to use NUL characters as separators between filenames instead of a newline, and -0 tells xargs to expect this. This prevents filenames with spaces, returns, or other weird characters from messing up your pipeline.

The -n 1 tells xargs to run a separate sed process for each "data.csv" file it finds, rather than trying to batch them together, which usually makes things more efficient. In this case, if you run

sed -n 2p file1 file2 file3

it'll internally concatenate all of its input files into one input stream, and print the second line of that. But RTFM: there may be a way to make sed not do that, that I've missed.

  • thanks for this solution but it still prints all results in a single line. – encodeflush Feb 1 '16 at 16:48
  • Is it possible that the files you're finding only have two lines, and don't end with a newline? – arensb Feb 2 '16 at 14:59
0

The result of the find is the list of filenames that sed needs to act upon. Thus use xargs to execute sed on that list, one by one. But because filenames can contains spaces and newlines even, better use the "-print0" option of find, to delimit the filenames with ascii zero. Also, when those files are big, you can save cpu-heat by stopping after line 2. And then you get:

find -name "data.csv" -print0 | xargs -0 -n 1 sed -n -e 2p -e 2q > final.csv
0

You can do the following:

find -name "data.csv" | xargs -n 1 sed -n 2p >> final.csv

Note above that using >> concatenates results from sed into final.csv, on a new line, instead of replacing them, whereas > simply replaces content in final.csv with output from sed.

  • thanks but prints the output in a single line – encodeflush Feb 1 '16 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.