0

I'm searching for a specific REGEX, 3 days I'm trying and trying but not founding the right answer.

I need to delete specific parts of an xml feed, I tried with sed, awk and it's not working right.

What I have :

...Something before
<description><![CDATA[Des  chercheurs de l&#x27;université de Columbia à New York ont mis au point un nouveau moyen de cacher un message dans un texte sans en altérer le sens et sans dépendre d&#x27;un format de fichier particulier. Nommée FontCode, cette idée est <a href="https://korben.info/cacher-des-informations-dans-un-texte-grace-a-des-modifications-sur-les-caracteres.html">Passage a la news suivante</a>]]></description>
... Other news

What I need :

...Something before
<description><![CDATA[Des  chercheurs de l&#x27;université de Columbia à New York ont mis au point un nouveau moyen de cacher un message dans un texte sans en altérer le sens et sans dépendre d&#x27;un format de fichier particulier.<a href="https://korben.info/cacher-des-informations-dans-un-texte-grace-a-des-modifications-sur-les-caracteres.html">Passage a la news suivante</a>]]></description>
... Other news
  • Select the multiples instances between "<\description></description>
  • Remove the last sentence which is not complete (before a href, "Nommée FontCode, cette idée est ")

Thank you for helping ! ;)

  • how do your outputs differ? – RomanPerekhrest Apr 9 '18 at 13:07
  • @RomanPerekhrest There's a fragment, Nommée FontCode, cette idée est , in the original. – Kusalananda Apr 9 '18 at 13:13
  • It can be anything. – w4rell Apr 9 '18 at 13:13
  • Here an other example : <description><![CDATA[Si vous êtes développeur et que vous cherchez un moyen rapide et facile de gérer vos packages NPM, voici Luna. Luna est une application à base d&#x27;Electron qui vous permettra de manipuler l&#x27;ensemble de vos packages npm. C&#x27;est à dire en installer de nouveaux, désinstaller <a href="korben.info/… a la news suivante</a>]]></description> – w4rell Apr 9 '18 at 13:15
  • The part to delete is between <a and the last correct sentence ( .?! character). – w4rell Apr 9 '18 at 13:17
0
sed -E '/^[[:blank:]]*<description><!\[CDATA\[/s/([!?.])[^!?.<]*<a/\1 <a/' file

This will look for all lines that start with the exact string <description><![CDATA[ (possibly preceded by spaces or tabs). On those lines, a substitution will be performed.

The substitution matches a sentence terminator ([!?.]) followed by any number of characters that are not sentence terminators or <, and the string <a. This will be replaced by the first sentence terminator, a space, and the string <a.

  • It doesn't seems to remove the last incorrect sentence. :/ I just tried on a file I can send for test. Thanks for this answer. – w4rell Apr 9 '18 at 14:06
  • @w4rell I'd be interested in seeing the example where it does not work. Do update the question with the relevant data. – Kusalananda Apr 9 '18 at 14:09
  • The file : hastebin.com/bovirocasi.xml The output is the same than the input. I used diff comand to compare these 2 files : diff -u feed.xml feed2.xml | sed -n '1,2d;/^[-+]/p' – w4rell Apr 9 '18 at 14:15
  • @w4rell Aha, the <description> tag does not occur at the start of the line. Drop ^ from the beginning of the sed expression. Or better yet, use /^[[:blank:]]*<description><!\[CDATA\[/. I have updated the answer. – Kusalananda Apr 9 '18 at 14:17
0

I Tested by below sed command and it worked fine for me

 sed "s/particulier\..*<a/particulier.<a/g" file

output

...Something before
<description><![CDATA[Des  chercheurs de l&#x27;université de Columbia à New York ont mis au point un nouveau moyen de cacher un message dans un texte sans en altérer le sens et sans dépendre d&#x27;un format de fichier particulier.<a href="https://korben.info/cacher-des-informations-dans-un-texte-grace-a-des-modifications-sur-les-caracteres.html">Passage a la news suivante</a>]]></description>
... Other news
  • Are you sure, this will remove anything that's not a sentence between : <description></description> – w4rell Apr 9 '18 at 14:09
0

While XML is a text-only format, trying to manipulate it with sed and/or awk is a terrible idea, because the format has so many corner cases and exceptions in the way it is used that you should really think of it as a binary format that just happens to be readable by the naked eye rather than a text format. It seems easy, until you actually try it. The short answer is, just, don't.

Instead, I would suggest the use of a scripting language that has a library for handling XML. There are many such libraries. In Perl, you could do something along these lines:

#!/usr/bin/perl -wCSDA
use strict;
use warnings;

package MyFilter;
use base qw(XML::SAX::Base);

sub new {
    my $class = shift;
    my @args = @_;
    my $self = $class->SUPER::new(@args);

    $self->{indesc} = 0;
    return $self;
}

sub start_element {
    my $self = shift;
    my $data = shift;
    if ($data->{LocalName} eq "description") {
        $self->{indesc} = 1;
    }
    return $self->SUPER::start_element($data);
}

sub end_element {
    my $self = shift;
    my $data = shift;
    if ($data->{LocalName} eq "description") {
        $self->{indesc} = 1;
    }
    return $self->SUPER::end_element($data);
}

sub characters {
    my $self = shift;
    my $data = shift;
    if(($self->{indesc}) == 1) {
        $data->{Data} =~ s/\.[^\.]*<a href/.<a href/;
    }
    return $self->SUPER::characters($data);
}

package main;

use XML::SAX::ParserFactory;
use XML::SAX::Writer;

my $writer = XML::SAX::Writer->new();

my $filter = MyFilter->new(Handler => $writer);

my $input = XML::SAX::ParserFactory->parser(Handler => $filter);

$input->parse_uri("input.xml");

This works as follows:

  • The package MyFilter; line signals a class which implements an XML::SAX filter:
    • sub new is the constructor, which really only creates the $self->{indesc} flag.
    • sub start_element gets called every time an XML element is opened. We check if the element in question is the <description> element; if so, we set the flag (and pass on further processing to the superclass).
    • sub end_element gets called every time an XML element is closed. We check if the element in question is the <description> element; if so, we clear the flag (and pass on further processing to the superclass).
    • sub characters gets called every time a text or CDATA element is processed. In that sub, we check if the flag is set; if it is, we apply a regular expression to the data passed on so that any incomplete sentence is dropped (only counting from a dot; improvement of this regex is left as an exercise to the reader ;-P)
  • The main package contains the starting point of the script:
    • It sets up an XML::SAX::Writer (which simply outputs the parsed XML data that it is passed in XML format again, to standard output by default), hooks that up to our filter (so the XML data passed to the writer contains the XML data that the filter received with the incomplete sentences removed), and hooks the filter up to an XML parser created with XML::SAX::ParserFactory.
    • The whole chain is then passed the input (it assumes it can be found in a file named input.xml).

If that seems complicated, that's because it is. If you have a choice, just say no to XML, and use something simpler instead, like JSON or YAML ;-)

  • Thank you for your complete answer, I learned a lot today ! :-) – w4rell Apr 9 '18 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.