3

I have a vertical file where each word (token) is on the separate line in 4 columns. There are also metastructures <doc>, <s>, ... The documents looks as follows:

<doc name="sth" url="http">
<p>
<s>
Here   here   k1gInSc1   here
is   be   k1gMnPc2   be
a  a   k2eAgMnPc1d1   a
sentence   sentence   k1gMnPc1   sentence
<g/>
.       .       kIx.
</s>
</p>
</doc>

the problem is that sometimes there is wrong encoding with characters as Ă or Ä in the first column, e.g.

<doc name="sth" url="http">
<p>
<s>
Here   here   k1gInSc1   here
is   be   k1gMnPc2   be
Ă  Ă   k?   Ă
sentence   sentence   k1gMnPc1   sentence
<g/>
.       .       kIx.
</s>
</p>
</doc>

I would need to find these characters and delete the whole document structure. So, if I find Ă on a line, I need to delete the whole content between <doc...> all lines </doc>.

My file has a billion lines and ca a few thousand lines contain wrong encoded characters.

I used grep to finding bad characters:

xzcat file.vert.xz | grep -i "Ă\|Ĺ\|ľ\|ş\|Ä" > file_bad_characters.txt

How can I detect these characters and delete not only the line but the whole text between <doc>structures.

  • Thank for your tips. Finally, I managed to solve the problem with the bad encoding, so I did not have to delete them. – Rodrigo Mar 13 '17 at 11:40
2

The right way to do this is to use a proper XML parser. However, in this case, the following might work as a workaround:

  1. Remove all blank lines from the file:

    sed -i '/^\s*$/d' file
    
  2. Add a blank line before each <doc>:

    sed -i 's/<doc/\n\n<doc/' file 
    
  3. User Perl's "paragraph mode" where "lines" are defined as "paragraphs" (sections of text preceded by an empty line):

    perl -00 -ne 'print unless /[ĂĹľşÄ]/' file > newfile
    

    Or, to make the replacements in the original file:

    perl -i.bak -00 -ne 'print unless /[ĂĹľşÄ]/' file
    

IMPORTANT: This assumes a well-structured file where everything is inside <doc... tags.

2

This isn't something you can do with just grep, I'm afraid; this requires that you retain some context about lines which grep can't provide. However, there are several other languages that can do this; here's an example with awk:

awk '/<doc>/ {text=""; output=1}
     /Ă|Ĺ|ľ|ş|Ä/{output=0}
     {text = text $0 "\n"}
     /<\/doc>/ {if(output==1){printf "%s", text}}"

This creates a buffer called text when we see a token <doc> in the input, and sets a flag to signal that we want to see the text printed; this flag is cleared when the forbidden characters are encountered. When we encounter the token </doc>, we check if the flag is still set; if so, we display the buffer. Finally, every line is added to our buffer, regardless of whether the flag is set or not.

  • This works and I used it many times. Thanks! – Rodrigo Sep 7 '17 at 7:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.