1

So i'm trying to write a simple script that'll parse a xml file and redirect output to a new file based off the name of the category it was found under. For example this is what the XML file looks like.

<category> Music </Category>
<url>https://www.youtube.com/watch?v=waAlgFq9Xq8</url>
<category> Movies </Category>
<url>https://www.youtube.com/watch?v=g4U4BQW9OEk</url>

My script looks something like this:

for i in *.xml; do
    name=$(grep -i "<category>" $i | awk '{print $1}')
    line=$(grep -i -A1 "<category>" $i)
    echo "$line" >> $filename
done

So for example Movies.log would contain all the links found under the Movies category, and Music.log would contain all the links found under the Music category.

2

Have you considered looping through each category? Like this:

for i in *.xml; do
    for category in $(sed -rn '/^<category>/{s/[^>]*> *([^ <]*).*/\1/p}' "$i"); do
        sed -rn "/^<category> *$category/,/^<category>/{s/<url> *([^ <]*).*/\1/p}" "$i" > "$category.log"
    done
done

Update: using awk

awk -v 'RS=<' -v 'cat=none' -F '>' \
'$1 ~ /^category$/ {gsub(/^ *| *$/,"",$2); cat=$2} \
$1 ~ /^url$/ {print $2 >> cat".log"}' \
*.xml
  • This avoids looping over the input files, and will append to the .log file for any category.

  • Using awk's record separator assignment -v 'RS=<' will mean that a category/url tag will be found anywhere (not just at the beginning of a line). A newline could occur anywhere in xml data.

  • Combining this with setting the field separator to '>', means that the first field of each record will be equivalent to an xml tag name.

  • Every time awk encounters a record where the first field is "category", a variable cat is set to the name for that category.

  • When awk encounters a record where the first field is "url" it will append that url to a file cat.log.

  • cat will be defined as none to begin with. This prevents failure in case a <url> is encountered without any preceding <category>.

  • The substitution gsub(/^ *| *$/,"",$2) is to remove the leading/trailing spaces for the category names that appear in your example input .xml file.


Note:

None of the above is foolproof. For proper xml input files an actual xml parser would be better – like xmlstarlet. But that would also require correctly formed xml files (the example input doesn't have matching <category> tags, for instance).

| improve this answer | |
0

I've prepared this solution:

grep -hP "<category.*>|<url.*>" *.xml | cut -d ">" -f 2 | cut -d "<" -f 1 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | gawk 'BEGIN { category = ""; } { if (!length($0)) { next; } if (length(category)) { printf("\necho -e \"%s\" >> \"%s.log\"", $0, category); category = ""; } else { category = $0; } } END { printf("\n"); }' | bash

It searches all .xml files in the current directory and appends the URL to the file named after the category found at the line preceding the URL (you can check the output by removing the | bash at the end).

Extract XML nodes only for the data we're interested in

By letting grep (for instance) search for a pattern in files named *.xml, we don't have to iterate over file names. Option -h to grep suppresses file names from the output. The pattern given to grep is a Perl-compatible regular expression (-P)

Extract values for the nodes we're interested in

Lines returned by the grep command obviously look like the below:

    <category> MyMusic </category>
    <url>https://www.youtube.com/watch?v=waAlgFq9Xq8123</url>
    <category> MyMovies </category>
    <url>https://www.youtube.com/watch?v=g4U4BQW9OEk456</url>
    <category>Music</category>
    <url>https://www.youtube.com/watch?v=waAlg</url>
    <category>              Music </category>
    <url>https://www.youtube.com/watch?v=waAlgFq9Xq8</url>
    <category> Movies </category>
    <url>https://www.youtube.com/watch?v=g4U4BQW9OEk</url>

we've just filtered out lines with data we don't need. Now we need to extract the values within the nodes, which boils down to extracting data between the opening and the closing tags, i.e. between signs > and < (we don't care which node it is, so we use a "generic" approach).

This can be easily achieved with | cut -d ">" -f 2 | cut -d "<" -f 1

which practically means take everything on the right side (-f 2) of sign > then, with the new result we have, take everything on the left side (-f 1) of sign <

Which leaves us with the below

 MyMusic 
https://www.youtube.com/watch?v=waAlgFq9Xq8123
 MyMovies 
https://www.youtube.com/watch?v=g4U4BQW9OEk456
Music
https://www.youtube.com/watch?v=waAlg
                Music 
https://www.youtube.com/watch?v=waAlgFq9Xq8
 Movies 
https://www.youtube.com/watch?v=g4U4BQW9OEk

Now we need to trim the values, here comes a small corrective step.

Trim the values

Trimming leading and trailing spaces with | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'

With -e, sed can execute scripts in the order they're given, without having to pipe an additional sed command (or multiple sed commands for other scenarios).

The first script passed to sed trims leading space (i.e. any [:space:] characters at the beginning of the string (@ each line)), while the second one trims trailing space (i.e. any [:space:] characters before the end of the string (@ each line).

Now we have something like the following, and we're nearly done:

MyMusic
https://www.youtube.com/watch?v=waAlgFq9Xq8123
MyMovies
https://www.youtube.com/watch?v=g4U4BQW9OEk456
Music
https://www.youtube.com/watch?v=waAlg
Music
https://www.youtube.com/watch?v=waAlgFq9Xq8
Movies
https://www.youtube.com/watch?v=g4U4BQW9OEk

Write out file append commands to stdout

Just like we write echo commands to append data to a file, we need something that automates the process. I chose to continue with gawk. gawk reads the data line by line, and grabs the category into a variable. When it reads another line, if the category variable is not empty, then the line contains the URL. Using this technique, we can simply issue commands like echo -e "current url" >> current_category.log

Notice the imperative usage of >> to append new data to the file. Using > will only write out the last URL, and we'll end up with a single line for every category!

As a result, we've just written data like the below to stdout:

echo -e "https://www.youtube.com/watch?v=waAlgFq9Xq8123" >> "MyMusic.log"
echo -e "https://www.youtube.com/watch?v=g4U4BQW9OEk456" >> "MyMovies.log"
echo -e "https://www.youtube.com/watch?v=waAlg" >> "Music.log"
echo -e "https://www.youtube.com/watch?v=waAlgFq9Xq8" >> "Music.log"
echo -e "https://www.youtube.com/watch?v=g4U4BQW9OEk" >> "Movies.log"

Pass data append commands to bash for execution

The last element in the pipeline, | bash ensures the echo commands are passed to bash for execution.

Note that gawk is capable of writing/appending data to files. But I intentionally wanted to have the smallest gawk script possible.

| improve this answer | |
-1
Tested with Below command and Worked fine

command

for i in `awk '/\<category\>/{print $2}' filename`; do sed -n '/'$i'/{n;p}' filename | awk -F ">" '{gsub(/<.*/,"",$2);print $2}' > $i.log; done

output

 cat Music.log 
https://www.youtube.com/watch?v=waAlgFq9Xq8

cat Movies.log 
https://www.youtube.com/watch?v=g4U4BQW9OEk
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.