Shell script or command to Strip text and Binary files excerpts from large Log Files

Question

I need to remove a large binary file (PDF file) from a large log file which is generated daily.This is seriously hogging space on our servers.I need to remove the large PDF from the logs to make the logs smaller and manageable

I need to take out the texts (or binary file) between the strings

• <my:PDF> and </my:PDF>
• <applicationForm> and </applicationForm>
• <image> and </image>
• <extractedSignature> and </extractedSignature>

I am not sure whether the sed utility can do this, these are large files and need to be pruned. I am not seeking log rotation advice, just a script or command that can strip these large logs of texts between the characters above. I am not sure how to do this. These files are rather large. I am not sure how to achieve this with sed , tail, head , tr or any other facility.

Answer 1

Sounds like your log files are actually an XML format. If that's the case, then what you REALLY should be doing is using an XML parser. Read this famous StackOverflow answer if you want some more info.

Based on your description, a SAX-based parser is probably your best bet: these are stream-oriented parsers and don't require you to load the whole file in memory. Since you're only performing a very simple transformation on the XML, XSLT is probably a really good candidate for this (and it should be FAST if you use xsltproc from libxslt). Here's an SO answer that demonstrates a potentially relevant technique.

Answer 2

If there are multiple lines between the /start/ and /end/ patterns then you can use ranges and d.

In general

/start/,/end/d

will delete all lines between /start/ and /end/ inclusive.

Create a file and add the following lines then save it as somefile

/<\"my:PDF\">/,/<\"\/my:PDF\">/d
/<\"applicationForm\">/,/<\"\/applicationForm\">/d
/<\"image\">/,/<\"\/image\">/d
/<\"extractedSignature\">/,/<\"\/extractedSignature\">/d

then use it like so

sed -f somefile logfile >prunedfile

Answer 3

You should be able to do something like this with sed:

cat LOG | sed -e "s/<\"my:PDF\">.*<\/\"my:PDF\">/<\"my:PDF\"><\/\"my:PDF\">/g;" > LOGx

Essentially this says "Look for the pattern <"my:PDF"> followed by any number of any character followed by the pattern </"my:PDF"> and replace it with the pattern <"my:PDF"></"my:PDF">, do this globally".

Repeat as needed for the other strings you need to substitute for.

Answer 4

Editing active logs is tricky and dangerous so you will have to wait until they are rotated out of service. If this is not possible you may need to check if you r logging program can pass the output to a filter rather than writing to a file. The filter can then strip the unwanted entries before the information is written to disk. This is somewhat more riskier as if the filter dies log entries will be lost.

Sadly sed regular expressions are "greedy". If a single log line has two or more fields you wish to strip out sed will strip out everything in between:

<"my:PDF">.*?<"/my:PDF"><"keeper">.*?<"/keeper"><"my:PDF">.*?<"/my:PDF">

With a -p option (see man perlrun) perl can be made to behave much like sed, but with the ability to ask for non-greedy regular expressions (the ? in *?):

perl -p -e '\
    s|<"my:PDF">.*?<"/my:PDF">||g; \
    s|<"applicationForm">.*?<"/applicationForm">||g; \
    s|<"image">.*?<"/image">||g; \
    s|<"extractedSignature">.*?<"/extractedSignature">||g; \
           '  huge.log >huge.log~ && \
mv huge.log~ huge.log