1

I have a series of text files with a consistent format like:

FirstName: Mary
LastName: Smith
Address: 123 Anywhere St
City: Nowhere
State: TX
Zip: 77777

I need to extract several lines from these files and output them into a csv file in a format like:

<filename>,<FirstName>,<City>,<Zip>

I can get the fields I want with a simple grep but I don't know how to get the output the way I need it.

  • 1
    What's your expected output – tachomi Aug 8 '16 at 21:14
  • Sounds. like a simple Perl task. What is between Zip and the next First Name? – waltinator Aug 8 '16 at 21:22
0

If you only have one record per file then this is a simple read loop.

#!/bin/bash

read_data()
{
  local first last addr city state zip

  file=$1

  while read -r header data
  do
    case $header in
      FirstName:) first=$data ;;
       LastName:) last=$data ;;
        Address:) addr=$data ;;
           City:) city=$data ;;
          State:) state=$data ;;
            Zip:) zip=$data ;;
               *) echo Ignoring bad line $header $data >&2
    esac
  done < $file
  echo "$file,$first,$last,$addr,$city,$state,$zip"
}

for file in *srcfiles*
do
  read_data $file
done

The read_data function reads each line and splits up the line into a "header" and a "data". Once we get to the end of the file we just print out the results.

We call that function once for each source file via the for loop.

Note some potential gotcha's: If there are commas in the data then this will break things, so you might want to do

  echo "\"$file\",\"$first\",\"$last\",\"$addr\",\"$city\",\"$state\",\"$zip\""

as the output to enclose everything inside "..." layout. If there's any " in the data then this may cause the CSV to be malformed as well.

Adjust the echo line to match the format you want.

0

Quick and dirty approach, may suit your requirements.

grep . *|perl -ne 'if(/FirstName: (.*)/){$f=$1}if(/City: (.*)/){$c=$1}if(/^(.*):Zip: (.*)/){print "$1,$f,$c,$2\n"}'

Example:

grep . *
f1.txt:FirstName: Mary
f1.txt:LastName: Smith
f1.txt:Address: 123 Anywhere St
f1.txt:City: Nowhere
f1.txt:State: TX
f1.txt:Zip: 77777
f2.txt:FirstName: Joe
f2.txt:LastName: Bloggs
f2.txt:Address: 444 Anywhere St
f2.txt:City: Nowhere2
f2.txt:State: TXA
f2.txt:Zip: 77737
grep . *|perl -ne 'if(/FirstName: (.*)/){$f=$1}if(/City: (.*)/){$c=$1}if(/^(.*):Zip: (.*)/){print "$1,$f,$c,$2\n"}'
f1.txt,Mary,Nowhere,77777
f2.txt,Joe,Nowhere2,77737
0

If there's a single record per file and you have GNU awk, you could do

gawk -F': +' -vOFS=, '
  BEGINFILE{delete rec}
  {rec[$1] = $2}
  ENDFILE{print FILENAME, rec["FirstName"], rec["City"], rec["Zip"]}
' file1.txt file2.txt ...
  • Doesn't the $2 mean that additional words will get lost? (eg "City: New York") would return "New". – Stephen Harris Aug 9 '16 at 1:15
  • @StephenHarris Doh! thanks - I have adjusted the field separator hopefully to fix that – steeldriver Aug 9 '16 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.