3

I have some Unix and Python experience, but am a bit rusty and don't know enough to figure this out myself.

I have a folder containing multiple files, from each of which I need to extract 2 specific strings and print them to a new text file with a tab between them. The files look like this:

mlu +t*CHI +f
Thu Jan 12 16:27:40 2017
mlu (08-Jan-2016) is conducting analyses on:  
  ONLY dependent tiers matching: %MOR;
****************************************
From file <adam01.cha>
MLU for Speaker: *CHI:
  MLU (xxx, yyy and www are EXCLUDED from the utterance and morpheme counts):
    Number of: utterances = 1236, morphemes = 2735
    Ratio of morphemes over utterances = 2.213
    Standard deviation = 1.300

I need to extract the file name and the value for "Ratio of morphemes over utterances". So I want to pull those strings out to create a new file that looks like this:

adam01.cha    2.213
adam02.cha    2.547
...

I can't figure out what exactly I need to do here. I'm not even sure where to start - I can't determine if egrep, awk, or sed is my best option, and how to put these into a for loop that would be able to iterate through the files properly.

4

You can use sed within a loop over each file in your current folder. You extract the relevant parts and append them via >> to a file named file like this:

for files in *; \
do sed -n -e '/^From file/ H;' \
          -e '/Ratio of morphemes over utterances/ {H; x; s/\n//g; s/From file <\(.*\)>.*Ratio of morphemes over utterances = \([0-9]*\.[0-9]*\).*/\1:    \2/g; p;}' "$files";
done >>file
  • loop: Because he said "I have a folder containing multiple files". why g: ok, this might be skipped, is equivalent imo. using H for both matches: Ok, maybe there's are more elegant solution – FloHe Jan 12 '17 at 23:19
  • Ok, might be possible and looking better. Personally I haven't used sed on multiple files before. Thanks for the hint. – FloHe Jan 12 '17 at 23:28
  • I removed the loop, if you don't mind. If you do, then feel free to revert. – xhienne Jan 13 '17 at 0:22
  • Thanks, but it seems to work not anymore in this way (if >1 file is in the folder). So I reverted it. – FloHe Jan 13 '17 at 0:26
  • @FloHe I used the version with the for loop and it worked perfectly. Thank you! (Now to parse it so I can better understand how to use sed for things like this in the future...) – Ed2122 Jan 13 '17 at 0:31
1
perl -0nE 'say "$1\t$2" if /From file <(.*?)>.*over utterances = (\d\S*)/s' * > out
1

Since you mentioned that you're familiar with Python, here's a python script that can do the job:

#!/usr/bin/env python
from __future__ import print_function
import os,re,sys

def read_file(filepath):
    with open(filepath) as fd:
         for line in fd:
             clean_line = line.strip()

             if 'From file' in clean_line:

                 words = re.split('<|>| ', clean_line)
                 print(words[-2],end=" ")

             if 'Ratio of morphemes over utterances' in clean_line:
                 print(clean_line.split('=')[-1])



def find_files(treeroot):
    selfpath = os.path.abspath(__file__)
    for dir,subdirs,files in os.walk(treeroot):
         for f in files: 
             filepath = os.path.abspath(os.path.join(dir,f))
             if selfpath  ==  filepath: continue
             try:
                 read_file(filepath)
             except IOError:
                 pass
def main():
    directory = '.'
    if len(sys.argv) == 2:
       directory = sys.argv[1]
    find_files(os.path.abspath(directory))

if __name__ == '__main__': main()

Sample run:

$ ./extract_data.py                                                                                               
adam02.cha  2.547
adam01.cha  2.213

The way this works is simple: we use os.walk to recursively walk through a directory, finding all files and excluding script itself, and for each file we run read_file() function , which reads each file line by line, and finds the appropriate fields. re.split() is used to break down the filename string into list of words more conveniently using space and < , and > as separators for words. The script can take command-line argument for a directory, but if it is not given - current working directory is assumed. That way you can run script given a path or from the directory where files are stored. As for creating new file with all the data, it is trivial - use shell's redirection as ./extract_data.py > /path/to/new_file.txt . Note of caution - redirect the script into a file that is located in different directory, since the new file might be enqueued into os.walk() and break the script. Additional improvement is that you could call the for loop for files as for f in sorted(files): to read files in sorted fashion.

1

You don't need a loop. Most text processing tools accept multiple arguments e.g. with sed:

sed '/From file/{s/.*<\(.*\)>/\1/;h
}
/Ratio of morphemes over utterances/!d
s/.*= //;H;x;s/\n/\t/' ./* > outfile

This extracts the file name and saves it to the hold buffer, deletes all lines except the ones where it extracts the "ratio", which it appends to the file name, then exchanges buffers and replaces the newline with a tab. Sure, \t is gnu sed specific so replace it with a literal tab (in terminal, press Ctrl+V then Tab) if you're not on a gnu setup. Even faster, with any sed:

sed '/From file/{s/.*<\(.*\)>/\1/;h
}
/Ratio of morphemes over utterances/!d
s/.*= //;H;x' ./* | paste - - > outfile

If your files are huge though, you might want to quit when on the second match (the line with the "ratio") and go to the next file. This is where gawk's nextfile comes in handy (I think it's POSIX but not sure which awk flavors support it...):

awk '/From file/{printf("%s\t", substr($3, 2, length($3)-2))}
/Ratio of morphemes over utterances/{print $7; nextfile}' ./* > outfile
0

you can try with awk command

awk '/Ratio of morphemes over utterances/{print FILENAME,$NF;next}' *.cha

if you want to extract the file name from the pattern From file < adam01.cha>

then, try the below awk command.

awk '/From file/{filename=$NF} filename && /Ratio of morphemes over utterances/{print FILENAME,$NF;filename="";next}' *.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.