Clean up concatenated XML files

Question

I'm trying to fix my Delicious export script to work with the new "trick" of only serving 1000 bookmarks at a time. The script concatenates results (XML) from multiple calls in a single file, and now I have to get rid of the header and footer text from the resulting file. Example:

<?xml ...
<posts ...
  <post ...
  ...          # 998 other posts
  <post ...
</posts>     # Line 1003
# The above lines are repeated N times before the final line:
</posts>

In other words, I want to remove any lines not starting with <post between (excluding) the third and the last line, and each chunk of XML is 1003 lines exactly, except for possibly the last one.

I'm guessing sed or awk should be perfect for this.

Answer 1

Because of the consistant format of the data, head and tail are your friends. This should even work for the last shorter file.

cat file | tail -n +3 | head -n -1 > trimmed_file

The tail -n +3 takes everything from the 3rd line down to the end of the file, and head -n -1 takes everything except the last line of the file.

Once you have a set of trimmed files, cat them together with an appropriate header and footer section for the whole file.

UPDATE: to avoid creating lots of extra files, just wrap it in a for loop:

for i in *
do
    cat $i | tail -n +3 | head -n -1 >> newfile
done

Get a template for the header by running one of the files through head to extract the first 3 lines, then run the for loop above. Then do a similar thing with tail to get the last line of one of the files and append it to newfile. I imagine you'll need to update the header and footer information.

Answer 2

Got it working:

sed -i -e '3,${/^</d}' file

In other words, between lines 3 and the last line, remove any line which starts with a <. Sorry the indentation didn't show up in the original post.

Answer 3

This should work

Lines containing <post : awk '/<post/ { print }'-

[jaypal~/Temp]$ cat xmlfile
<?xml ...
<posts ...
  <post ...
  ...          # 998 other posts
  <post ...
</posts>     # Line 1003
# The above lines are repeated N times before the final line:
</posts>

[jaypal~/Temp]$ cat xmlfile | awk '/<post/ { print }'
<posts ...
  <post ...
  <post …

Lines starting with <post : awk '/^<post/ { print }'-

[jaypal~/Temp]$ cat xmlfile | awk '/^<post/ { print }'
<posts …

You don't even need the { print } statement. That is implicit in AWK anytime the pattern is TRUE.

So this would work as well -

[jaypal~/Temp]$ cat xmlfile | awk '/^<post/'
<posts ...

Answer 4

This looks a bit clunky. Why not process the data as it comes in?

bookmarks_count=$chunk_size
total_bookmarks_count=0
{
  while [ $bookmarks_count -eq $chunk_size ]; do
    chunk=$(wget … -O - "$EXPORT_URL?start=$total_bookmarks_count")
    bookmarks_count=$(printf %s "$chunk" | grep -c "$bookmark_prefix")
    total_bookmarks_count=$((total_bookmarks_count + bookmarks_count))
    printf %s "$chunk" |
    sed -e 's#><#>\n<#g' -e "$EXPORT_COMPATIBILITY" -e "$EXPORT_COMPATIBILITY"
  done
  echo '<\/posts>'
} >"$EXPORT_PATH"

You can even avoid storing each chunk in memory, though it's a bit trickier. Here's a method that only works in ksh and zsh; in other shells, the right-hand side of the pipeline runs in a subshell so the value of total_bookmarks_count is not updated.

{
  total_bookmarks_count=0
  while
      wget … -O - "$EXPORT_URL?start=$bookmarks_count" |
      sed -e … |
      tee /dev/fd/3 |
      this_chunk_size=$(grep -c "$bookmark_prefix")
      [[ $this_chunk_size = $chunk_size ]]
  do
    ((total_bookmarks_count += chunk_size))
  done
  echo '<\/posts>' >&3
} 3>"$EXPORT_PATH"

Here's a way to make this method work in other shells, where the only information you can get out of a pipeline is its return status.

: >"$EXPORT_PATH"
total_bookmarks_count=0
while
    wget … -O - "$EXPORT_URL?start=$bookmarks_count" |
    sed -e … |
    tee -a "$EXPORT_PATH" |
    [ $(grep -c "$bookmark_prefix") = $chunk_size ]
do
  total_bookmarks_count=$((total_bookmarks_count + chunk_size))
done
echo '<\/posts>' >> "$EXPORT_PATH"