I have a folder with 179 tab-separated text files. They all have the same format. Here's a link to sample data:


(This sample has the 5 headers, 5 lines of data I want, and all of the trailing data that I do not want.)

I want to strip the first four header lines completely, then use the fifth as the header for all rows in a single file that combines the data - WITHOUT the trailing data.

I also want to append two fields: a filename field, and one for the value of Prelim 4. So the output would look like:

Filename    Curr    Storefront Name Identifier  M-Type  Quantity    Net Amt Net Amt Total   Code    Title   Item    A-Type  B-Type  Vendor_ID   Offline Indicator   company donotuse
file1   ZAR ZA  44056971    Family Monthly PAID 1   .061038 .061038 US3E40400141    subtitle1   person  1   1   US3E40400141    1   BlueMelon   
file2   ZAR ZA  44056975    Family Monthly PAID 1   .061038 .061038 US3E40400142    subtitle2   person2 1   1   US3E40400142    1   BlueMelon   
file3   ZAR ZA  44057007    Family Monthly PAID 1   .061038 .061038 US3E40400144    subtitle3   person3 1   1   US3E40400144    1   BlueMelon   
file179 ZAR ZA  44057330    Family Monthly PAID 1   .061038 .061038 US3E40400107    subtitle5   person5 1   1   US3E40400107    1   BlueMelon

I'm a COMPLETE noobie to shell scripting. Literally 1 week of experience.

To add the first field, I've gotten this far, and am stuck:

gawk 'NR==5;FNR>5 {print FILENAME,"\t",$0} /Row/ {f=0; nextfile} f' *.txt > result.txt

I have no idea how to add the second field (the value of Prelim Data 4).

Any help? Thanks!

  • An example would make it much easier to understand what you want to do. Please post mock data and the output you'd like to obtain from it. Commented Aug 15, 2017 at 19:26
  • "I also want to append two fields: a filename field, and one for the value of Prelim 4" But "Prelim 4" is supposed to have two fields... Commented Aug 15, 2017 at 19:50
  • I just added a link to sample/mockup data. mvabl.com/public/sample_data.txt Also, Hauke, thank you. I wasn't clear. the "two fields" in Prelim 1-4 are key-value pairs. So I want to keep the value from Prelim4. Should be clearer if you look at the mockup data I just linked to.
    – Steve
    Commented Aug 15, 2017 at 19:54

2 Answers 2

awk 'FNR == 4 { pd4=$2; };
     NR == 5 { print $0 " FILENAME PD4";};
     FNR == 6 { print $0 " " FILENAME " " pd4; nextfile; }' file1 file2
  • Thanks Hauke! Would you be able to explain the FNR == 7? Should that be FNR == 6?
    – Steve
    Commented Aug 15, 2017 at 20:11
  • NR is counting through all files. FNR is reset for every new file. In your first example the data was in line 7. Commented Aug 15, 2017 at 20:14
  • Thanks. Do you know how I can strip out the "trailing data"? In each file, that section starts with a row whose first field is "Row Count". I don't want that data.
    – Steve
    Commented Aug 15, 2017 at 20:28
  • @Steve My code should not have output anything after the 6th/7th line. But I have added a command to explicitly skip the rest of the file. Commented Aug 15, 2017 at 20:55

Since you appear to have GNU awk, how about

gawk '
  /^Row Count/ {nextfile} 
  NR == 5 {$0 = "Filename" OFS $0; print} 
  FNR > 5 {$0 =  FILENAME  OFS $0; print}
' OFS='\t' *.txt
  • Yup, that worked! Thanks, steel driver. I tried to +1 you, but I don't have enough points to have it show up publicly.
    – Steve
    Commented Aug 15, 2017 at 21:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .