0

I would like to extract variables from a text file with Bash, Perl and Regex.

The file looks like this (and is already read into the variable $str):

Filename: XXXXX
Type: XXX
Size: XXXX
Unimportant thing: XXXX

Filename: YYYYY
Type: YYY
Size: YYYY
Unimportant thing: YYYY

I need Filename, Type and Size for each block. An array would be the best but a strings containing these variables separated by a given character is also acceptable.

However, sometimes some of the fields (e.g. Size or Type) is missing. I would like to omit these records, so I think I need a Regex which can match through multiple lines.

I tried the following:

perl -pe 's/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n/\1\t\2\t\3\n/' <<< $str

but this printed out the original text without modification.

Then I tried it without the p commandline parameter (I hoped this way the whole file will be processed instead of iterating lines):

perl -e 's/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n/\1\t\2\t\3\n/' <<< $str

This one did not print anything (empty result).

Then I tried adding print in front of the Regex because I thought maybe the removal of -p caused that Perl does not know that I want the results printed:

perl -e 'print s/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n/\1\t\2\t\3\n/' <<< $str

Still no success (empty result).

What am I missing?

Update:

I would like this as a one line perl command.

  • You're using a one-liner on the command line so it looks like you want that in an answer (as opposed to a regular, multi-line script). You should mention that explicitly if it's what you want. (Along with any other requirements you have.) – B Layer Aug 25 '17 at 8:05
  • @BLayer: Does it make a real difference? I don't know much about Perl but I had the impression that one-liners can do anything that more-liners even if they become a bit long. – z32a7ul Aug 25 '17 at 8:10
  • Yeah, but when it starts exceeding more than a couple lines of code it gets really ugly and ought to be in a separate script, really. IMO. – B Layer Aug 25 '17 at 8:10
  • Case in point, the (new) solution I just posted. A little long for my taste. :) Hopefully it works for you, though. – B Layer Aug 25 '17 at 8:45
  • @z32a7ul: You don't reveal much about what variations in your input data can occur. If the fields are always in the same sequence, and if you are sure that the filenames do not contain any white space, you can do it simply by fmt -999 <inp.txt|grep 'Filename:.*Type:.*Size:'|cut -d ' ' -f 1-6. This assumes that your input data is stored in inp.txt. – user1934428 Aug 25 '17 at 12:34
1

My Perl knowledge is thin but since no one else has provided a Perl answer I'll give it a go.

Pass your data in as file and it will print tab-separated lines with three values per line:

perl -e 'while (<>) { $s .= $_; } chomp $s; @arr = split(/\n{2,}/, $s); foreach my $a(@arr) { $a =~ s/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n.*/$1\t$2\t$3\n/ || next; print "$a"; } ' infile

Result:

XXXXX   XXX     XXXX
YYYYY   YYY     YYYY

It's a bit brute-force but works by splitting the input up into paragraphs/blocks and then applies your multi-line regex to each.

Details...

  • while (<>) { $s .= $_; } - Slurp the input into a single string.
  • chomp $s - Remove trailing newline from the string.
  • @arr = split(/\n{2,}/, $s) - Split string on consecutive newlines. This breaks it up into paragraphs/blocks. Store the blocks in an array.
  • foreach my $a(@arr) - Loop over each array element (block). The next two lines of code are applied to each block.
  • $a =~ s/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n.*/$1\t$2\t$3\n/ || next - Extract values from the three fields of interest. If no substitution occurs (meaning the regex doesn't match because, for example, a value is missing) then skip this block and move to the next one.
  • print "$a" - Print the result of the substitution: the three values separated by tabs.

Again, I don't use much Perl so there probably are more elegant solutions than this.

  • Thanks for your answer but I don't see how this solution would address the problem that some records have missing fields (e.g. size or type) and should be omitted. That's why I tried multiline matching. – z32a7ul Aug 25 '17 at 7:50
  • Oops, I kinda missed that part...let me fix that. – B Layer Aug 25 '17 at 7:51
  • Looks promising but at first try it left the unimportant rows in the output unlike your first solution. In that case, was the chomp thing that removed those lines? – z32a7ul Aug 25 '17 at 10:39
  • @z32a7ul Not sure what you mean. In my tests it only returns three columns, ignoring "Unimportant thing" and it doesn't return rows for any blocks that have one or more missing values. (BTW, the output is the same as for the sed solution in the other answer here.) – B Layer Aug 25 '17 at 11:21
  • 1
    Thanks for your answer. Although it is not the final solution, it helped me a lot to start searching. I ended up with this: local tsv=$(perl -e 'while( <> ) { $w .= $_; } while( $w =~ /Filename: ([^\n]*)\n[\S\s]*?Type: ([^\n]*)\n[\S\s]*?Size: ([^\n]*)\n/g ) { print "$1\t$2\t$3\n"; }' <<< str) – z32a7ul Aug 26 '17 at 12:07
0

Not a big expert in perl, but with sed it will look this way:

sed  -n '/^$/d;/^Filename/,/^Unimportant/{:a;/Unimportant/!{N;ba};s/Filename: \([^\n]*\)\nType: \([^\n]*\)\nSize: \([^\n]*\)\n.*/\1\t\2\t\3/p};'

Where:

  • /^$/d -- will remove all empty lines
  • /^Filename/,/^Unimportant/ will match each block from Filename to Unimportant separately. I'm assuming you have Unimportant record in every block.
  • :a;/Unimportant/!{N;ba}; will concatenate the entire block in one buffer. Is needed as sed is not able to work with multiline regexes or process several lines in a time in other way
  • s/Filename: \([^\n]*\)\nType: \([^\n]*\)\nSize: \([^\n]*\)\n.*/\1\t\2\t\3/p}; will do the replacement to the format you need (based on your perl regex)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.