Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm trying to fix my Delicious export script to work with the new "trick" of only serving 1000 bookmarks at a time. The script concatenates results (XML) from multiple calls in a single file, and now I have to get rid of the header and footer text from the resulting file. Example:

<?xml ...
<posts ...
  <post ...
  ...          # 998 other posts
  <post ...
</posts>     # Line 1003
# The above lines are repeated N times before the final line:
</posts>

In other words, I want to remove any lines not starting with <post between (excluding) the third and the last line, and each chunk of XML is 1003 lines exactly, except for possibly the last one.

I'm guessing sed or awk should be perfect for this.

share|improve this question
    
So do you need lines only with <posts or <posts and <post –  JS웃 Nov 25 '11 at 0:25
add comment

4 Answers

Because of the consistant format of the data, head and tail are your friends. This should even work for the last shorter file.

cat file | tail -n +3 | head -n -1 > trimmed_file

The tail -n +3 takes everything from the 3rd line down to the end of the file, and head -n -1 takes everything except the last line of the file.

Once you have a set of trimmed files, cat them together with an appropriate header and footer section for the whole file.

UPDATE: to avoid creating lots of extra files, just wrap it in a for loop:

for i in *
do
    cat $i | tail -n +3 | head -n -1 >> newfile
done

Get a template for the header by running one of the files through head to extract the first 3 lines, then run the for loop above. Then do a similar thing with tail to get the last line of one of the files and append it to newfile. I imagine you'll need to update the header and footer information.

share|improve this answer
    
I was hoping to avoid any temporary files, which is why I asked how to do it inline. –  l0b0 Nov 24 '11 at 22:42
    
for i in *; do cat $i | tail -n +3 | head -n -1 >> newfile; done –  dr-jan Nov 24 '11 at 22:50
    
Won't this pass strings at each whitespace?? –  JS웃 Nov 25 '11 at 0:26
    
Jaypal - I apologise, but I don't understand your comment. –  dr-jan Nov 25 '11 at 22:08
    
Hi @dr-jan, what I mean't was if the line contains a white space string eg. this post, then for loop, instead of using the whole line will pass this first and then post. –  JS웃 Nov 26 '11 at 7:14
show 1 more comment
up vote 1 down vote accepted

Got it working:

sed -i -e '3,${/^</d}' file

In other words, between lines 3 and the last line, remove any line which starts with a <. Sorry the indentation didn't show up in the original post.

share|improve this answer
add comment

This should work

Lines containing <post : awk '/<post/ { print }'-

[jaypal~/Temp]$ cat xmlfile
<?xml ...
<posts ...
  <post ...
  ...          # 998 other posts
  <post ...
</posts>     # Line 1003
# The above lines are repeated N times before the final line:
</posts>

[jaypal~/Temp]$ cat xmlfile | awk '/<post/ { print }'
<posts ...
  <post ...
  <post …

Lines starting with <post : awk '/^<post/ { print }'-

[jaypal~/Temp]$ cat xmlfile | awk '/^<post/ { print }'
<posts …

You don't even need the { print } statement. That is implicit in AWK anytime the pattern is TRUE.

So this would work as well -

[jaypal~/Temp]$ cat xmlfile | awk '/^<post/'
<posts ...
share|improve this answer
add comment

This looks a bit clunky. Why not process the data as it comes in?

bookmarks_count=$chunk_size
total_bookmarks_count=0
{
  while [ $bookmarks_count -eq $chunk_size ]; do
    chunk=$(wget … -O - "$EXPORT_URL?start=$total_bookmarks_count")
    bookmarks_count=$(printf %s "$chunk" | grep -c "$bookmark_prefix")
    total_bookmarks_count=$((total_bookmarks_count + bookmarks_count))
    printf %s "$chunk" |
    sed -e 's#><#>\n<#g' -e "$EXPORT_COMPATIBILITY" -e "$EXPORT_COMPATIBILITY"
  done
  echo '<\/posts>'
} >"$EXPORT_PATH"

You can even avoid storing each chunk in memory, though it's a bit trickier. Here's a method that only works in ksh and zsh; in other shells, the right-hand side of the pipeline runs in a subshell so the value of total_bookmarks_count is not updated.

{
  total_bookmarks_count=0
  while
      wget … -O - "$EXPORT_URL?start=$bookmarks_count" |
      sed -e … |
      tee /dev/fd/3 |
      this_chunk_size=$(grep -c "$bookmark_prefix")
      [[ $this_chunk_size = $chunk_size ]]
  do
    ((total_bookmarks_count += chunk_size))
  done
  echo '<\/posts>' >&3
} 3>"$EXPORT_PATH"

Here's a way to make this method work in other shells, where the only information you can get out of a pipeline is its return status.

: >"$EXPORT_PATH"
total_bookmarks_count=0
while
    wget … -O - "$EXPORT_URL?start=$bookmarks_count" |
    sed -e … |
    tee -a "$EXPORT_PATH" |
    [ $(grep -c "$bookmark_prefix") = $chunk_size ]
do
  total_bookmarks_count=$((total_bookmarks_count + chunk_size))
done
echo '<\/posts>' >> "$EXPORT_PATH"
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.