Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to remove a large binary file (PDF file) from a large log file which is generated daily.This is seriously hogging space on our servers.I need to remove the large PDF from the logs to make the logs smaller and manageable

I need to take out the texts (or binary file) between the strings

  • <my:PDF> and </my:PDF>
  • <applicationForm> and </applicationForm>
  • <image> and </image>
  • <extractedSignature> and </extractedSignature>

I am not sure whether the sed utility can do this, these are large files and need to be pruned. I am not seeking log rotation advice, just a script or command that can strip these large logs of texts between the characters above. I am not sure how to do this. These files are rather large. I am not sure how to achieve this with sed , tail, head , tr or any other facility.

share|improve this question
    
Does your server/app have an option for lower level of logging? I guess the above only show up for DEBUG log level. –  alex Jul 13 '11 at 11:18
1  
Do not cross-post. Now you have partially overlapping answers on three sites. Flagged for merge from Stack Overflow and Server Fault to Unix. –  Gilles Jul 13 '11 at 12:22

4 Answers 4

If there are multiple lines between the /start/ and /end/ patterns then you can use ranges and d.

In general

/start/,/end/d

will delete all lines between /start/ and /end/ inclusive.

Create a file and add the following lines then save it as somefile

/<\"my:PDF\">/,/<\"\/my:PDF\">/d
/<\"applicationForm\">/,/<\"\/applicationForm\">/d
/<\"image\">/,/<\"\/image\">/d
/<\"extractedSignature\">/,/<\"\/extractedSignature\">/d

then use it like so

sed -f somefile logfile >prunedfile
share|improve this answer

Editing active logs is tricky and dangerous so you will have to wait until they are rotated out of service. If this is not possible you may need to check if you r logging program can pass the output to a filter rather than writing to a file. The filter can then strip the unwanted entries before the information is written to disk. This is somewhat more riskier as if the filter dies log entries will be lost.

Sadly sed regular expressions are "greedy". If a single log line has two or more fields you wish to strip out sed will strip out everything in between:

<"my:PDF">.*?<"/my:PDF"><"keeper">.*?<"/keeper"><"my:PDF">.*?<"/my:PDF">

With a -p option (see man perlrun) perl can be made to behave much like sed, but with the ability to ask for non-greedy regular expressions (the ? in *?):

perl -p -e '\
    s|<"my:PDF">.*?<"/my:PDF">||g; \
    s|<"applicationForm">.*?<"/applicationForm">||g; \
    s|<"image">.*?<"/image">||g; \
    s|<"extractedSignature">.*?<"/extractedSignature">||g; \
           '  huge.log >huge.log~ && \
mv huge.log~ huge.log
share|improve this answer
    
many thanks to you all ,i used another approach sed -n .. –  Doknik Aug 4 '11 at 9:35

You should be able to do something like this with sed:

cat LOG | sed -e "s/<\"my:PDF\">.*<\/\"my:PDF\">/<\"my:PDF\"><\/\"my:PDF\">/g;" > LOGx

Essentially this says "Look for the pattern <"my:PDF"> followed by any number of any character followed by the pattern </"my:PDF"> and replace it with the pattern <"my:PDF"></"my:PDF">, do this globally".

Repeat as needed for the other strings you need to substitute for.

share|improve this answer

Sounds like your log files are actually an XML format. If that's the case, then what you REALLY should be doing is using an XML parser. Read this famous StackOverflow answer if you want some more info.

Based on your description, a SAX-based parser is probably your best bet: these are stream-oriented parsers and don't require you to load the whole file in memory. Since you're only performing a very simple transformation on the XML, XSLT is probably a really good candidate for this (and it should be FAST if you use xsltproc from libxslt). Here's an SO answer that demonstrates a potentially relevant technique.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.