-1

I have a folder containing multiple files, and from each file I need to 1) extract all lines containing items from a prespecified list of words; 2) pre-pend the file name to that line 3) separate the extracted line into two separate strings, deleting some content in between.

This is the content of the files I need to extract from:

freq +fchi +t*CHI +s"m|v"
Fri Feb 24 10:24:17 2017
freq (08-Jan-2016) is conducting analyses on:
  ONLY speaker main tiers matching: *CHI;
    and those speakers' ONLY dependent tiers matching: %MOR;
****************************************
From file <sarah004.cha>
Speaker: *CHI:
  2 v|eye-3S
  1 v|get
  2 v|get&PAST
  1 v|go&PAST
  1 v|help
  2 v|ride
  3 v|see
  1 v|toe-3S
------------------------------
    8  Total number of different item types used
   13  Total number of items (tokens)
0.615  Type/Token ratio

An example of the list of words I'm looking for is see|know|look (actual list is much longer, about 25 words).

What I want to end up with is this:

sarah004.cha    3    see
sarah006.cha    3    know
sarah010.cha    1    look
sarah010.cha    2    see
...

So in addition to extracting lines containing the specified words, I'd also be separating the number and the word and deleting the v|.

Some files in the directory may not contain any of the words on the list, others may contain many of the words on the list.

I'm not sure if I can accomplish this with sed; I tried this as a starting point (just extracting any lines with counts) and it wouldn't finish running:

sed '/From file/{s/.*<\(.*\)>/\1/;h
}
/^  [0-9] v|/!d
s/.*= //;H;x' ./* | paste - - > sarahverbcounts.txt

I would be fine doing this in multiple steps, although I'm sure it could be accomplished with one command/script and I just don't know how. I think perl could handle this, but I don't know the syntax (I tried using a perl example I found and couldn't figure out how to properly change it for my purposes). Maybe I need try this in Python, which would allow me to specify the word list, but I am not familiar with how to direct Python to iterate over all files in a directory one by one and have them all write to the same output file (I've looked at examples but I don't quite understand them). Any advice is appreciated (and, if possible, explanations of how your solution works would be wonderful, because I'm new enough to this that I can't always parse the helpful answers that I'm given).

  • It looks like you want something like awk 'NR==FNR {a["v|"$0]=$0; next} $2 in a {print FILENAME, $1, a[$2]}' wordlist *.cha (where wordlist is a file containing the search words, one per line) - but I'm not sure based on your description – steeldriver Feb 24 '17 at 17:30
1

Put the list of words to search for in a file called words.txt and then run:

grep -Hf words.txt files.* | sed 's/:/ /;s/v|/ /'
  • This worked! It uses the names of the files that the strings are extracted from, which has a different extension than the .cha file name contained in the file itself, but that's easily replaced. – Ed2122 Feb 26 '17 at 15:56
1

You can do it many ways as shown. The patterns are stored, 1 per line in a list file.

Note: We distinguish the patterns file from the remaining data files (presumably your *cha files) by different stratagems in bash, perl, & sed. In case of "sed" the patterns are stored in a dot file, aka hidden files, & appended with a "." to assist "sed" code in distinguishing a pattern file from a regular data file. This is a WA for sed as it does not have as in awk the concept of "FNR".

Grep

grep -E -f ./.your_patterns_listfile ./* |
sed -e 's/://' -e 's/v|//'

Perl

find . -maxdepth 1 -type f -name '*' -exec \
   perl -wMstrict -Mvars='*pat' -lne '
      BEGIN { ($pat) = @ARGV; }
      if ( $pat ne $ARGV ) {
         @pat = map { quotemeta } keys %pat unless @pat;
         next unless /^From file [<]/ .. /^--*/;
         my $cha;
         /^From file [<]([>]*)[>]/ and $cha = $1;
         for my $pat ( @pat ) {
            /^\s+ (\d+) \s+ v[|] ($pat) $/x and print(join("\t",$ARGV,$1,$2)),last
         }
      } else {
         $pat{$_}++;
      }
   ' ./your_patterns_listfile {} +

Sed

echo '.' >> ./.your_patterns_listfile
find . -maxdepth 1 -type f -name '*' -exec \
   sed -e '
      1{
         :pats
            N
         /\n\.$/!bpats
         s///;h;d
      }

      /^From file </!d
      $d;N
      /\n[ ]*\([1-9][0-9]*\)[ ]v|/{
         s//\n\1\t|/;s/$/|/
         G;s/\n/&&/;s/$/\n/
         /\n\n[1-9][0-9]*\t|\([^|]*\)|.*\n\1\n/{
            s/^From file <\([^>]*\)>\n\n\([1-9][0-9]*\)\t|\([^|]*\)|/\1\t\2\t\3\n&/
            P;s/\n\n.*//;D
         }
      }
      s/\n.*//;s/^/\n/;D
   ' ./.your_patterns_listfile {} +

Bash

find . -type f -name '*' ! -name 'your_patterns_listfile' -exec \
   sh -c '
      shift $1
      flag=
      eval "`echo '\''readonly NL=qsq'\'' | tr '\''qs'\'' '\''\047\012'\''`"; # newline
      pats=$(< $1); shift
      cat "$@" |
      while IFS= read -r line
      do
         case $line in
            "From file <"* )
               cha=${line#*"<"}
               cha=${cha%">"}
               unset flag
               continue
               ;;

            *[0-9]*v\|* )
               pat=${line#*"|"}
               num=${line%%"v"*}
               num=${num//[ ]/}
               case $pats in
                  "$pat" | *"$NL$pat$NL"* | "$pat$NL"* | *"$NL$pat" )
                     ${flag+":"} printf "%s\t" "$cha" "$num" "$pat"
                     ${flag+":"} echo
                     ;;
               esac
               ;;

            '' | *[!-]* )
               :
               ;;

            * )
               flag=
               ;;
         esac
      done
   ' 2 1 ./your_patterns_listfile {} +
  • Thank you for all of these options and your explanation. – Ed2122 Feb 26 '17 at 15:57
-1

Maybe something like this:

egrep '^\s+[0-9]+\s+v\|([0-9a-zA-Z-]+)$' files* | sed -e 's/v|//g'
  • Where's the prespecified list of words? – Wildcard Feb 25 '17 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.