Need to extract 2 strings from specific lines of multiple files and print to new file, separated by tab

Question

I have some Unix and Python experience, but am a bit rusty and don't know enough to figure this out myself.

I have a folder containing multiple files, from each of which I need to extract 2 specific strings and print them to a new text file with a tab between them. The files look like this:

mlu +t*CHI +f
Thu Jan 12 16:27:40 2017
mlu (08-Jan-2016) is conducting analyses on:  
  ONLY dependent tiers matching: %MOR;
****************************************
From file <adam01.cha>
MLU for Speaker: *CHI:
  MLU (xxx, yyy and www are EXCLUDED from the utterance and morpheme counts):
    Number of: utterances = 1236, morphemes = 2735
    Ratio of morphemes over utterances = 2.213
    Standard deviation = 1.300

I need to extract the file name and the value for "Ratio of morphemes over utterances". So I want to pull those strings out to create a new file that looks like this:

adam01.cha    2.213
adam02.cha    2.547
...

I can't figure out what exactly I need to do here. I'm not even sure where to start - I can't determine if egrep, awk, or sed is my best option, and how to put these into a for loop that would be able to iterate through the files properly.

Answer 1

You can use sed within a loop over each file in your current folder. You extract the relevant parts and append them via >> to a file named file like this:

for files in *; \
do sed -n -e '/^From file/ H;' \
          -e '/Ratio of morphemes over utterances/ {H; x; s/\n//g; s/From file <\(.*\)>.*Ratio of morphemes over utterances = \([0-9]*\.[0-9]*\).*/\1:    \2/g; p;}' "$files";
done >>file

Answer 2

perl -0nE 'say "$1\t$2" if /From file <(.*?)>.*over utterances = (\d\S*)/s' * > out

Answer 3

Since you mentioned that you're familiar with Python, here's a python script that can do the job:

#!/usr/bin/env python
from __future__ import print_function
import os,re,sys

def read_file(filepath):
    with open(filepath) as fd:
         for line in fd:
             clean_line = line.strip()

             if 'From file' in clean_line:

                 words = re.split('<|>| ', clean_line)
                 print(words[-2],end=" ")

             if 'Ratio of morphemes over utterances' in clean_line:
                 print(clean_line.split('=')[-1])



def find_files(treeroot):
    selfpath = os.path.abspath(__file__)
    for dir,subdirs,files in os.walk(treeroot):
         for f in files: 
             filepath = os.path.abspath(os.path.join(dir,f))
             if selfpath  ==  filepath: continue
             try:
                 read_file(filepath)
             except IOError:
                 pass
def main():
    directory = '.'
    if len(sys.argv) == 2:
       directory = sys.argv[1]
    find_files(os.path.abspath(directory))

if __name__ == '__main__': main()

Sample run:

$ ./extract_data.py                                                                                               
adam02.cha  2.547
adam01.cha  2.213

The way this works is simple: we use os.walk to recursively walk through a directory, finding all files and excluding script itself, and for each file we run read_file() function , which reads each file line by line, and finds the appropriate fields. re.split() is used to break down the filename string into list of words more conveniently using space and < , and > as separators for words. The script can take command-line argument for a directory, but if it is not given - current working directory is assumed. That way you can run script given a path or from the directory where files are stored. As for creating new file with all the data, it is trivial - use shell's redirection as ./extract_data.py > /path/to/new_file.txt . Note of caution - redirect the script into a file that is located in different directory, since the new file might be enqueued into os.walk() and break the script. Additional improvement is that you could call the for loop for files as for f in sorted(files): to read files in sorted fashion.

Answer 4

You don't need a loop. Most text processing tools accept multiple arguments e.g. with sed:

sed '/From file/{s/.*<\(.*\)>/\1/;h
}
/Ratio of morphemes over utterances/!d
s/.*= //;H;x;s/\n/\t/' ./* > outfile

This extracts the file name and saves it to the hold buffer, deletes all lines except the ones where it extracts the "ratio", which it appends to the file name, then exchanges buffers and replaces the newline with a tab. Sure, \t is gnu sed specific so replace it with a literal tab (in terminal, press Ctrl+V then Tab) if you're not on a gnu setup. Even faster, with any sed:

sed '/From file/{s/.*<\(.*\)>/\1/;h
}
/Ratio of morphemes over utterances/!d
s/.*= //;H;x' ./* | paste - - > outfile

---

If your files are huge though, you might want to quit when on the second match (the line with the "ratio") and go to the next file. This is where gawk's nextfile comes in handy (I think it's POSIX but not sure which awk flavors support it...):

awk '/From file/{printf("%s\t", substr($3, 2, length($3)-2))}
/Ratio of morphemes over utterances/{print $7; nextfile}' ./* > outfile

Answer 5

you can try with awk command

awk '/Ratio of morphemes over utterances/{print FILENAME,$NF;next}' *.cha

if you want to extract the file name from the pattern From file < adam01.cha>

then, try the below awk command.

awk '/From file/{filename=$NF} filename && /Ratio of morphemes over utterances/{print FILENAME,$NF;filename="";next}' *.txt