I have a large text file with columns A, B, C, and D. How do I read the unique values for columns C and D and write those unique rows to a regular text file or in JSON format?

  • 2
    Please post a row so I can see the dorm of the text for process by regular expression. Oct 2 '14 at 19:54
  • 3
    As Tyler said, please show us an example of your input and desired output. Do both C and D need to be unique? Do you mean unique combinations of C+D? Would foo bar be the same as bar foo?
    – terdon
    Oct 2 '14 at 19:58
  • A B C D q a ENT123 New York b c ENT123 New York d e ENT345 New Jersy f n ENT345 New Jersy x y ENT346 New Hampshire w p ENT346 New Hampshire file looks like this and below is the expected result Oct 3 '14 at 14:07
  • C D ENT123 New York ENT345 New Jersy ENT346 New Hampshire Oct 3 '14 at 14:10
  • 3
    Edit your examples into your question: they are too hard to read as comments.
    – jasonwryan
    Oct 6 '14 at 6:58

Try this script:

sort -k 3,5 -u yourFile.txt | awk '{output = $3; for(i=4; i<=NF; i++) output = output " " $(i); print output;}'

where option -k sets columns C and D as key, -u unique and awk prints columns C and D.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.