1

I am looking to clean (edit in place) files that look like this

<id>474488</id>
<name>Shawn</name>
<nr>143385</nr>

<id>474490</id>
<name>Bob</name>
<nr>.27. 43-88</nr> # this is the line of interest

<id>474568</id>
<name>Jim</name>
<nr>
</nr> # sometimes there will be no value and a closing tag on a newline, this can be ignored
....

desired output:

<id>474488</id>
<name>Shawn</name>
<nr>143385</nr>

<id>474490</id>
<name>Bob</name>
<nr>274388</nr> # note that nr data has been cleaned to digits only

<id>474568</id>
<name>Jim</name>
<nr>
</nr>
....

In other words, I want to delete all non-digit characters/strip out certain characters from the data enclosed in <nr> </nr> tags.

my code:

sed -Ee  '/<nr>/ s/>(.*)</>\1</g' test1.txt

What this does:

  • selects only lines with <nr> in them

  • replaces tags and content in them (by content of capturing group 1 = no changes because I don't know how to process the content of capturing group 1).

Also, ideally, I would not want to replace > <, but lookarounds (to tell sed to start after > and before <) seem impossible in sed.

What I need to add (but don't know how):

Filter the contents of capturing group 1 (either strip out . and - and whitespace or allow digits only) before inserting them.

How do I do this?

Do I need to use a different tool?

6
  • Can we safely assume the opening and closing <nr> and </nr> tags will always be on the same line? – terdon Apr 3 at 17:17
  • No, that's why the last batch of the example is on a new line. – shilaoing Apr 3 at 17:19
  • Ah, duh yes. Sorry, I missed that. Presumably though you can also have cases where you do have a value and a newline, right? Or will you only have newlines if you have no value? Also, is this XML? Can you use a proper XML parser instead of naive text-editing tools? – terdon Apr 3 at 17:22
  • I did observe the newline with empty values but I don't currently know if newline + value will appear. Sorry. Yes, it's XML but I have never dealt with XML and I figured basic linux tools should do (might be wrong). I have no use for XML in any way and will probably have to convert later anyway. – shilaoing Apr 3 at 17:29
  • Basic Linux tools usually work on "lines", so whenever you need to deal with embedded newlines, things get scary. Also, obligatory reading: SO Q&A on regexes for X/HTML and here's what it looks like if you really want to do it right manually: If this is XML, can you post a more complete example so we can use a dedicated parser (see Shell script to remove child xml tags conditionally for an example). – terdon Apr 3 at 17:33
2

This looks like it's a fragment of XML. Adding an enclosing <root/> element so we have below XML, we can then use an XML editing tool,

xmlstarlet ed -u '//nr' -x 'translate(text(), "- .", "")' file.xml
<?xml version="1.0"?>
<root>
  <id>474488</id>
  <name>Shawn</name>
  <nr>143385</nr>
  <id>474490</id>
  <name>Bob</name>
  <nr>274388</nr>
  <id>474568</id>
  <name>Jim</name>
  <nr>
</nr>
</root>

The important piece here is the XPath translate() function. It's similar in operation to the UNIX/Linux command tr in that it replaces characters from one string with those from the other (the first parameter is the value on which to operate).

I've used nr as the hook on which to operate. The element path can be more precise if required (in my example /root/nr could be used too).

Very few filter tools actually work on files in-place. They write a temporary file and then use that to replace the original. In this instance we have to implement this ourselves

xmlstarlet ... file.xml >file.xml.tmp && mv -f file.xml.tmp file.xml
2
  • Thanks. So 'ed' doesn't do edits in place but applies edits to stdout only? – shilaoing Apr 10 at 10:09
  • Almost nothing actually does edits in place – roaima Apr 10 at 11:08
0

If this is a proper structured language like XML, you really should be using dedicated parsers instead (consider xmlstarlet, for instance). That said, as long as your files are small enough to fit in memory, and assuming that as you say in the comments, you will only ever have a newline within the field in cases where the field is empty so no substitution is necessary, you could actually just do this:

$ sed '/<nr>/{s/[. -]*//g}' file
<id>474488</id>
<name>Shawn</name>
<nr>143385</nr>

<id>474490</id>
<name>Bob</name>
<nr>274388</nr>#thisisthelineofinterest

<id>474568</id>
<name>Jim</name>
<nr>
</nr>
....

For more complex cases, and if I could not use a proper parser, I would go for perl:

$ perl -nle '$k=1 if /<nr>/; if($k){s/[. -]//g}; $k=0 if /<\/nr>/; print' file
<id>474488</id>
<name>Shawn</name>
<nr>143385</nr>

<id>474490</id>
<name>Bob</name>
<nr>274388</nr>#thisisthelineofinterest

<id>474568</id>
<name>Jim</name>
<nr>
</nr>
....

But note that the above will fail if you can have multiple tags on the same line, like this:

<nr>143385</nr><name>Shawn - Mary</name>

In such cases, the - will also be removed from the value of <name>. This sort of edge case is why a parser really is your best bet.

For both the perl and sed options, you can use -i to edit the file in place:

sed -i '/<nr>/{s/[. -]*//g}' file
perl -i -nle '$k=1 if /<nr>/; if($k){s/[. -]//g}; $k=0 if /<\/nr>/; print' file
3
  • Thanks. What do the curly braces do in the sed example? – shilaoing Apr 3 at 18:42
  • @shilaoing habit, really. They're not needed here since you only want to do one operation when the line matches /<nr>/, but they would allow you to group multiple ones together: /foo/{s/foo/bar/; s/bar/baz/}. – terdon Apr 3 at 18:44
  • While it's not as selective as the approach I had intended (remove characters within tags only rather than remove characters from the whole line), the sed approach solved my problem and did not require me to learn new languages such as xml or perl. Thank you for this. – shilaoing Apr 10 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.