I have a file which contains 12 Million lines each line is broken into 8 parts all of which are separated by ";". File1 Example: AAA;BBB;CCC;DDD;EEE;FFF;GGGXX;HHH

Each line has a unique ID in $7. I have another file (File2) which has ~20K of these unique IDs listed as: GGGXX; GGGXY; GGGXZ;

Each on its own line.

I need the result to create a new file with the complete data row from File1 based on the IDs in File2.

I cannot seem to get it to work when File2 has more than 1 entry using the code below.

awk -F ";" 'NR==FNR{a[$1]} ($7 in a){print}' File2.txt File1.txt

Is there something wrong with how the IDs are listed in File2 or is there something else that I am doing wrong?

  • Are you aware that 'F should be -F? Is that just a typo? – glenn jackman Nov 30 '18 at 19:34
  • This is not a bug, but it's a bit sloppy: a[$21++] -- that creates an array key for the value field 21 (the value for that array key is 0 or the empty string), then it increments the field 21 value. The ++ has no effect on the array at all and can be removed. You probably intended a[$21]++, but it's not necessary that the array has a value for that key. – glenn jackman Nov 30 '18 at 19:36
  • Having said that, you do not actually ask a question in your question. Please read How do I ask a good question? – glenn jackman Nov 30 '18 at 19:38
  • @glennjackman Thanks for your help. I had made some typos. What I am trying to do is have both files compared and as I said in the post, File 1 has all of my data and File 2 $1 has the small group of data that I need to pull out of File 1 $7. When I try this and have File 2 with only 1 data point, it works, But when I have all 20K data points, I am not given any data from File1. I'll edit the main post with more information. – Leo P Nov 30 '18 at 21:06

Kindly use below command to get the required output.

awk  -F "," 'NR==FNR {a[$7];next}($1 in a){print $1}' file1 file2 > newfile
  • It seems that it does not really work the way i needed it to. The issue is that it still does not work when File2 has more than 1 entry. The only way it works is if File 2 has only 1 ID in the txt file. Another issue is that this way only prints out the line of File2 whereas I want to print the matching line in File1 that has the same ID as in File2. Thanks – Leo P Dec 3 '18 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.