1

I have several files formatted like this:

...
<title> Field1 : DATA_FIELD_1</title>
...
<i class="blablabla"></i> <b>Field2 : </b> <span>DATA_FIELD_2</span>
...
<i class="blablabla"></i> <b>Field3 : </b> <span>DATA_FIELD_3</span>
...
<i class="blablabla"></i> <b>Field4 : </b> <span>DATA_FIELD_4</span >
...
<i class="blablabla"></i> <b>Field5 : </b> <span>DATA_FIELD_5 </span>
...

I would like to read each file, and get the data for each field, and write it into a new file in a pipe-separated format.

For instance:

FileID | Field1 | Field2 | Field3 | Field4 | Field5
1 | DATA_FIELD_1 | DATA_FIELD_2 | DATA_FIELD_3 | DATA_FIELD_4 | DATA_FIELD_5
2 | DATA_FIELD_1 | DATA_FIELD_2 | DATA_FIELD_3 | DATA_FIELD_4 | DATA_FIELD_5
3 | DATA_FIELD_1 | DATA_FIELD_2 | DATA_FIELD_3 | DATA_FIELD_4 | DATA_FIELD_5
4 | DATA_FIELD_1 | DATA_FIELD_2 | DATA_FIELD_3 | DATA_FIELD_4 | DATA_FIELD_5

i managed to extract the data improperly with grep using the command :

grep -o 'Field1 : .*\|Field2 : .*\|Field3 : .*\|Field4 : .*\|Field5 : .*' File-* >> NewFile

but this is the result i get 

File-1:Field1 : DATA_FIELD_1</title>
File-1:Field2 : </b> <span>DATA_FIELD_2</span>
File-1:Field3 : </b> <span>DATA_FIELD_3</span>
File-1:Field4 : </b> <span>DATA_FIELD_4</span >
File-1:Field5 : </b> <span>DATA_FIELD_5 </span>
File-2:Field1 : DATA_FIELD_1</title>
File-2:Field2 : </b> <span>DATA_FIELD_2</span>
File-2:Field3 : </b> <span>DATA_FIELD_3</span>
File-2:Field4 : </b> <span>DATA_FIELD_4</span >
File-2:Field5 : </b> <span>DATA_FIELD_5 </span>
File-3:Field1 : DATA_FIELD_1</title>
File-3:Field2 : </b> <span>DATA_FIELD_2</span>
File-3:Field3 : </b> <span>DATA_FIELD_3</span>
File-3:Field4 : </b> <span>DATA_FIELD_4</span >
File-3:Field5 : </b> <span>DATA_FIELD_5 </span>
Improve this question
5
  • That's already something. Add this to your answer and tell us exactly what your problem is. What is your actual problem with the result you get? What do you feel you're stuck with? – lgeorget Mar 29 '17 at 21:04
  • up to this point,"" grep -o 'Field1 : .*\|Field2 : .*\|Field3 : .*\|Field4 : .*\|Field5 : .*' Unsaved >> NewFile " can extract data but, they are written in lines prepended by the file name – user223423 Mar 29 '17 at 21:06
  • Can you add all this to your question? Well documented attempts show provious research and attract more answers. – lgeorget Mar 29 '17 at 21:07
  • thank you; it worked ... but only for files with and id between 1 and 9 when i used it with all files... i have about 500 000 – user223423 Mar 29 '17 at 22:34
  • That's because I match only one character to identify the file. Fixed! – lgeorget Mar 29 '17 at 22:42
0

Generally, it's a bad idea to process HTML/XML with regular expressions because they are not expressive enough to handle all corner cases. However, your grep output suggests that we can in your precise case.

Here is a way to process the output you get from your grep command with sed:

sed -e 'N;N;N;N;s/^File-\(.*\):Field1 : \(.*\)<\/title>.*Field2 : .*<span>\(.*\)<\/span *>.*Field3 : .*<span>\(.*\)<\/span *>.*Field4 : .*<span>\(.*\)<\/span *>.*Field5 : .*<span>\(.*\)\s*<\/span *>/\1 | \2 | \3 | \4 | \5 | \6/'

Explanations:

  • N;N;N;N;: this is used to join lines five by five (in order to have all fields from a file in the same line)
  • s/: this starts a substitution command, the first part is the match, the second one the replacement
  • ^File-\(.*\):: this matches the file number
  • Field1 : \(.*\)<\/title> : this matches the first field
  • .*Field2 : .*<span>\(.*\)<\/span *> : this matches the second one
  • ...
  • /\1 | \2 | \3 / \4 | \5 | \6/: this is the replacement part: each group delimited by \(.*\) in the first part is captured and reusable through special variables \1, \2, etc.

Result:

1 | DATA_FIELD_1 | DATA_FIELD_2 | DATA_FIELD_3 | DATA_FIELD_4 | DATA_FIELD_5 
2 | DATA_FIELD_1 | DATA_FIELD_2 | DATA_FIELD_3 | DATA_FIELD_4 | DATA_FIELD_5 
3 | DATA_FIELD_1 | DATA_FIELD_2 | DATA_FIELD_3 | DATA_FIELD_4 | DATA_FIELD_5
Improve this answer
0

Unother possible approach is to use a XML processor. In this case XMLstarlet.

xmlstarlet sel -t -v 'substring-after(//title,":")' \
                  -m //span -o "|" -v .  \
               -t -n  *.xml

Where:

  • sel -t -- select XML parts acording to the following template
  • -v 'substring-after(//title,":")' -- is the value of title after ":"
  • -m //span -o "|" -v . -- matches span and prints "|" and its value
  • -t -n -- add a newline
Improve this answer
1
  • this doesn't seem to work with the generated file, since xmlstarlet keeps complaining about a missing start tag "<"... and if applied to the orginal files, wich are php files it keeps complaining about some undefined entities and premature end of files in tags – user223423 Apr 1 '17 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.