0

I have a large text file with multiple occurrences of a tag containing a URL:

[tag]https://example.com/222389/link/11835457224168404[/tag]

I need to reformat the tags as follows:

[new-tag]11835457224168404[/new-tag]

(capture just the part of the url after 'link' (the 'id') and modify the tag to 'new-tag':

  • There can be multiple tags per line;
  • The tag locations are not uniform - they are found in random positions throughout the file;
  • The tag content can have a space at the start (' http'), use 'http://" or 'https://' and sometimes use 'www';
  • The tag occasionally has content or space at the end (after the 'id') such as follows:

    [tag]https://example.com/222389/link/11835457224168404/qwertyiop[/tag]
    

    or

    [tag]https://example.com/222389/link/11835457224168404?link=11835457224168401    [/tag]
    
  • There are sometimes occurrences of '[tag]' on their own (without the closing [/tag] or 'http') that need to be ignored.

How can I do this with sed or alternatives?

  • Are "tags" always numbers? If not how can you tell the tag in example.com/222389/link/11835457224168404/qwertyiop is not the qwertyiop? If a "[tag]" is on its own is to be ignored can we assume that when it is not to be ignored the "[/tag]" is on the same line? – icarus May 1 at 4:20
  • Desired format: [tag]id[/tag]. The id is always the group of digits after '/link/' e.g. 11835457224168404. '/link/' is a constant. Sometimes that id (the group of digits) is followed by text that needs to be removed e.g. '/qwertiop' or ?link=xyz. – Michael T May 1 at 4:27
  • ^^ Yes, the target is any occurrence of opening tag followed by a closing tag - i.e. [tag]URL[/tag] but not a solo [tag] or [/tag]. There can be multiple tags on one line (e.g. "lorem ipsum [tag]URL[/tag] dolor sit amet, [tag]URL[/tag] consectetur adipiscing elit [tag]"). – Michael T May 1 at 4:35
  • Are the tags really in square brackets? If they were XML tags, you could use XML parsing tools like xmlstarlet. This would make thi processing of the data much more robust. – Kusalananda May 1 at 10:23
0

Strategy

Whilst it is possible to write regular expressions which don't match multi-character strings, they get complicated. We will use a trick to convert [tag] and [/tag] into single characters and then use negated character classes. In this script I will use control-a and control-b. It is critical that these characters don't appear in the file. As these are hard to type, I will use a couple of variables s and e for the start and end tags. I use notend to represent any sequence of characters which is not the end tag.

#!/bin/bash
s=$'\001' # control-a, for the start tag 
e=$'\002' # control-b, for the end tag
notend="[^$e]*" # expression for not the end tag.
# Program, Change the tags into single characters
# change matched pairs of tags into new form
# convert any unmatched tags back to original form
prog='
s:\[tag]:'"$s"':g
s:\[/tag]:'"$e"':g
s:'"$s$notend"'/link/\([0-9]*\)'"$notend$e"':[new-tag]\1[/newtag]:g
s:'"$s"':[tag]:g
s:'"$e"':[/tag]:g'

# run sed, passing any parameters  
sed -e "$prog" "$@"

usage

Save this script, make it executable, then run it giving the datafile as a parameter and redirecting the output to a temp file. examine the temp file.

| improve this answer | |
  • Superb! The script works perfectly - thank you so much. – Michael T May 1 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.