How to find specific csv files and extract their second row into a single csv file

Question

I'm trying to find out how to filter specific csv files by name and then extract their second row into a single csv file. I'm not sure whether it is a good idea to find files and then pipe it with commands like sed/awk to extract desired line.

find -name "data.csv" | sed -n 2p > final.csv 

update

Using a for loop in bash script generates desired output but prints all output in a single line.

for OUTPUT in $(find -name "data.csv")
do

        sed -n 2p $OUTPUT

done

Please help me with your suggestions.

Answer 1

If you look at the output of find -name "data.csv", you'll see that it prints the full pathnames of files named data.csv. The pipeline passes that on to sed, which prints the second line of its input.

So what you're saying is, "Here's a list of files. Give me the second one one the list." What you really want to say is "Here's a list of files. For each one, give me the second line." And for that, you want to use xargs.

Most likely, you'll need

find -name "data.csv" -print0 | xargs -0 -n 1 sed -n 2p > final.csv

It's often a good idea to use find -print0 | xargs -0: the -print0 causes find to use NUL characters as separators between filenames instead of a newline, and -0 tells xargs to expect this. This prevents filenames with spaces, returns, or other weird characters from messing up your pipeline.

The -n 1 tells xargs to run a separate sed process for each "data.csv" file it finds, rather than trying to batch them together, which usually makes things more efficient. In this case, if you run

sed -n 2p file1 file2 file3

it'll internally concatenate all of its input files into one input stream, and print the second line of that. But RTFM: there may be a way to make sed not do that, that I've missed.

Answer 2

The result of the find is the list of filenames that sed needs to act upon. Thus use xargs to execute sed on that list, one by one. But because filenames can contains spaces and newlines even, better use the "-print0" option of find, to delimit the filenames with ascii zero. Also, when those files are big, you can save cpu-heat by stopping after line 2. And then you get:

find -name "data.csv" -print0 | xargs -0 -n 1 sed -n -e 2p -e 2q > final.csv

Answer 3

You can do the following:

find -name "data.csv" | xargs -n 1 sed -n 2p >> final.csv

Note above that using >> concatenates results from sed into final.csv, on a new line, instead of replacing them, whereas > simply replaces content in final.csv with output from sed.