Extract variables from text file into array with Bash, Perl and Regex

Question

I would like to extract variables from a text file with Bash, Perl and Regex.

The file looks like this (and is already read into the variable $str):

Filename: XXXXX
Type: XXX
Size: XXXX
Unimportant thing: XXXX

Filename: YYYYY
Type: YYY
Size: YYYY
Unimportant thing: YYYY

I need Filename, Type and Size for each block. An array would be the best but a strings containing these variables separated by a given character is also acceptable.

However, sometimes some of the fields (e.g. Size or Type) is missing. I would like to omit these records, so I think I need a Regex which can match through multiple lines.

I tried the following:

perl -pe 's/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n/\1\t\2\t\3\n/' <<< $str

but this printed out the original text without modification.

Then I tried it without the p commandline parameter (I hoped this way the whole file will be processed instead of iterating lines):

perl -e 's/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n/\1\t\2\t\3\n/' <<< $str

This one did not print anything (empty result).

Then I tried adding print in front of the Regex because I thought maybe the removal of -p caused that Perl does not know that I want the results printed:

perl -e 'print s/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n/\1\t\2\t\3\n/' <<< $str

Still no success (empty result).

What am I missing?

Update:

I would like this as a one line perl command.

Answer 1

My Perl knowledge is thin but since no one else has provided a Perl answer I'll give it a go.

Pass your data in as file and it will print tab-separated lines with three values per line:

perl -e 'while (<>) { $s .= $_; } chomp $s; @arr = split(/\n{2,}/, $s); foreach my $a(@arr) { $a =~ s/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n.*/$1\t$2\t$3\n/ || next; print "$a"; } ' infile

Result:

XXXXX   XXX     XXXX
YYYYY   YYY     YYYY

It's a bit brute-force but works by splitting the input up into paragraphs/blocks and then applies your multi-line regex to each.

Details...

• while (<>) { $s .= $_; } - Slurp the input into a single string.
• chomp $s - Remove trailing newline from the string.
• @arr = split(/\n{2,}/, $s) - Split string on consecutive newlines. This breaks it up into paragraphs/blocks. Store the blocks in an array.
• foreach my $a(@arr) - Loop over each array element (block). The next two lines of code are applied to each block.
• $a =~ s/Filename: ([^\n]*)\nType: ([^\n]*)\nSize: ([^\n]*)\n.*/$1\t$2\t$3\n/ || next - Extract values from the three fields of interest. If no substitution occurs (meaning the regex doesn't match because, for example, a value is missing) then skip this block and move to the next one.
• print "$a" - Print the result of the substitution: the three values separated by tabs.

Again, I don't use much Perl so there probably are more elegant solutions than this.

Answer 2

Not a big expert in perl, but with sed it will look this way:

sed  -n '/^$/d;/^Filename/,/^Unimportant/{:a;/Unimportant/!{N;ba};s/Filename: \([^\n]*\)\nType: \([^\n]*\)\nSize: \([^\n]*\)\n.*/\1\t\2\t\3/p};'

Where:

• /^$/d -- will remove all empty lines
• /^Filename/,/^Unimportant/ will match each block from Filename to Unimportant separately. I'm assuming you have Unimportant record in every block.
• :a;/Unimportant/!{N;ba}; will concatenate the entire block in one buffer. Is needed as sed is not able to work with multiline regexes or process several lines in a time in other way
• s/Filename: \([^\n]*\)\nType: \([^\n]*\)\nSize: \([^\n]*\)\n.*/\1\t\2\t\3/p}; will do the replacement to the format you need (based on your perl regex)