I'm trying to fix my Delicious export script to work with the new "trick" of only serving 1000 bookmarks at a time. The script concatenates results (XML) from multiple calls in a single file, and now I have to get rid of the header and footer text from the resulting file. Example:

<?xml ...
<posts ...
  <post ...
  ...          # 998 other posts
  <post ...
</posts>     # Line 1003
# The above lines are repeated N times before the final line:

In other words, I want to remove any lines not starting with <post between (excluding) the third and the last line, and each chunk of XML is 1003 lines exactly, except for possibly the last one.

I'm guessing sed or awk should be perfect for this.

  • So do you need lines only with <posts or <posts and <post – jaypal singh Nov 25 '11 at 0:25

Because of the consistant format of the data, head and tail are your friends. This should even work for the last shorter file.

cat file | tail -n +3 | head -n -1 > trimmed_file

The tail -n +3 takes everything from the 3rd line down to the end of the file, and head -n -1 takes everything except the last line of the file.

Once you have a set of trimmed files, cat them together with an appropriate header and footer section for the whole file.

UPDATE: to avoid creating lots of extra files, just wrap it in a for loop:

for i in *
    cat $i | tail -n +3 | head -n -1 >> newfile

Get a template for the header by running one of the files through head to extract the first 3 lines, then run the for loop above. Then do a similar thing with tail to get the last line of one of the files and append it to newfile. I imagine you'll need to update the header and footer information.

  • I was hoping to avoid any temporary files, which is why I asked how to do it inline. – l0b0 Nov 24 '11 at 22:42
  • for i in *; do cat $i | tail -n +3 | head -n -1 >> newfile; done – dr-jan Nov 24 '11 at 22:50
  • Won't this pass strings at each whitespace?? – jaypal singh Nov 25 '11 at 0:26
  • Jaypal - I apologise, but I don't understand your comment. – dr-jan Nov 25 '11 at 22:08
  • Hi @dr-jan, what I mean't was if the line contains a white space string eg. this post, then for loop, instead of using the whole line will pass this first and then post. – jaypal singh Nov 26 '11 at 7:14

Got it working:

sed -i -e '3,${/^</d}' file

In other words, between lines 3 and the last line, remove any line which starts with a <. Sorry the indentation didn't show up in the original post.


This looks a bit clunky. Why not process the data as it comes in?

  while [ $bookmarks_count -eq $chunk_size ]; do
    chunk=$(wget … -O - "$EXPORT_URL?start=$total_bookmarks_count")
    bookmarks_count=$(printf %s "$chunk" | grep -c "$bookmark_prefix")
    total_bookmarks_count=$((total_bookmarks_count + bookmarks_count))
    printf %s "$chunk" |
    sed -e 's#><#>\n<#g' -e "$EXPORT_COMPATIBILITY" -e "$EXPORT_COMPATIBILITY"
  echo '<\/posts>'

You can even avoid storing each chunk in memory, though it's a bit trickier. Here's a method that only works in ksh and zsh; in other shells, the right-hand side of the pipeline runs in a subshell so the value of total_bookmarks_count is not updated.

      wget … -O - "$EXPORT_URL?start=$bookmarks_count" |
      sed -e … |
      tee /dev/fd/3 |
      this_chunk_size=$(grep -c "$bookmark_prefix")
      [[ $this_chunk_size = $chunk_size ]]
    ((total_bookmarks_count += chunk_size))
  echo '<\/posts>' >&3

Here's a way to make this method work in other shells, where the only information you can get out of a pipeline is its return status.

    wget … -O - "$EXPORT_URL?start=$bookmarks_count" |
    sed -e … |
    tee -a "$EXPORT_PATH" |
    [ $(grep -c "$bookmark_prefix") = $chunk_size ]
  total_bookmarks_count=$((total_bookmarks_count + chunk_size))
echo '<\/posts>' >> "$EXPORT_PATH"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.