0

Say I have a large CSV file with a header and several columns. For the purpose of this question I will consider a small file with just two columns. We can call it use_rep.

user_id,rep
885,500K+
22565,200K+
7453,200K+
86440,100K+
116858,100K+
22222,100K+
38906,100K+
10762,<100K
70524,<100K

I'd like to send each row to a file corresponding to the value on the second column. For example, I'd like there to be a file whose name is 200K+ and whose content is

user_id,rep
22565,200K+
7453,200K+

The contents of use_rep should not be assumed to be ordered in anyway. The pattern to be used would ideally accept regular expressions.

No sed or perl is preferred.

  • I think AWK can do this easily, but I don't really know how. – regex Apr 11 at 10:44
3

Ignoring the header (which you can tack on later):

awk -F, 'NR > 1 {print > $2}' use_rep

which will print each line to a file named by the second column:

~ head *[0-9]*
==> 100K+ <==
86440,100K+
116858,100K+
22222,100K+
38906,100K+

==> 200K+ <==
22565,200K+
7453,200K+

==> 500K+ <==
885,500K+

==> <100K <==
10762,<100K

To put the header, maybe something like:

awk -F, 'NR == 1 {header = $0; next} # save header, skip this line
  !a[$2]++ { print header > $2 } # print if second field hasnt been seen before 
  { print > $2 }' use_rep

Result:

~ head *[0-9]*
==> 100K+ <==
user_id,rep
86440,100K+
116858,100K+
22222,100K+
38906,100K+

==> 200K+ <==
user_id,rep
22565,200K+
7453,200K+

==> 500K+ <==
user_id,rep
885,500K+

==> <100K <==
user_id,rep
10762,<100K
70524,<100K
  • I'm having some issues with this due to commas inside text identifiers (" "). Is there an easy fix? – regex Apr 11 at 13:51
  • Not with awk. You should use a tool with support for quoted csv, like csvkit or Python or Perl. – muru Apr 11 at 14:13
0

Tried with below command and it worked fine

command

for i in `awk -F "," '{print $2}' l.txt | sort | uniq`; do sed -n '/'$i'$/p' l.txt > $i.txt ; done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.