extract file to generate and rename differenf files

Question

I have a csv (in which firt and second column are separeted by ",") file like this:

Column1,Column2
4e,info1
4t,info2
45t,info3
3,info4

I would like to obtained 4 different file, one for each row and the file names are taken from the Column1 while the content from the Column2.

my expected output are:

filename1 = 4e.smi

info1

filename2 = 4t.smi

info2

filename3 = 45t.smi

info3

filename4 = 3.smi

info4

I think that I could generate two different variable, one related to the first column and the other to the second one and use this variable to create new file doing this in loop for all the rows. But I tried this cmd line but it didn't work:

while IFS=',' read -r name smile; do write "$smile" "$name".smi; done < InputFIle.txt

Could someone help me please with this challange?

Thanks.

Answer 1

Your loop is almost correct apart from the fact that the write command is not the command that you want to use. Instead, use printf with a redirection. You also need to make sure to skip the initial line, which we can do in a number of different ways. Below, I'm using tail -n +2, but you could also use sed 1d.

tail -n +2 InputFIle.txt |
while IFS=, read -r name string; do
    printf '%s\n' "$string" >"$name".smi
done

Note that this assumes that the filename is written to only once, as if you have the same value in the first column on two or more lines, the later lines would cause the data already written to the file be overwritten. You may therefore possibly want to change > to >> to append to the output file. This change would require you to additionally delete the files if you want to run the code twice (or you'll get duplicated data in the output).

A possibly more efficient way to do this would be to use awk like so:

awk -F, 'NR > 1 { print $2 >($1 ".smi") }' InputFIle.txt

This prints the second comma-delimited field to the filename given by the first field. It skips the initial line by testing NR (the number of records read so far) against 1.

This does not suffer from the same issue as the shell loop. The output file would be truncated (emptied or created) upon the first print to it, and subsequent outputs would then be appended.

If you have further fields on some lines, the awk variant would need to be modified to print all fields apart from the first:

awk -F, 'NR > 1 { name = $1; sub("[^,]*,",""); print >(name ".smi") }' InputFIle.txt

This saves the first field in a separate variable, name, and then removes that field from the original line with sub(). It then prints the remaining line to the file.

Answer 2

This will work efficiently using any tail+sort+awk no matter how many output files need to be generated:

tail -n +2 file | sort | awk -F, '$1!=prev{close(out); out=$1".smi"; prev=$1} {print $2 > out}'

If you don't close the output files as you go then you can run into a "too many open files" error with most awks, or get a significant slow down from GNU awk as it tries to manage opening/closing the files for you, once you get past a threshold which I've seen hit at under 20 output files. We're sorting first so we don't have to keep opening/closing the output files every time a duplicate $1 is seen throughout the input.