Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

The 5GB files I have are streams of data rows formed:

    {datarow1...},{datarow2...},...,{datarowN...}

so actually could say that there are lines {}, and even line separators, but coming as a three char sequence: },{

I want to do two things:

  1. print "lines" that have string "error" in it:

    grep -o -P {[^{}]+?error.+?} ES01.log > ES01.err.log
    
  2. make the file more "friendly" by explicitly producing files with new line separators

    <ES01.log sed -e 's/},{/}\n{/g' > ESnl01.log
    

While the above works for relatively small files (up to ~100MB), my files are unfortunately a lot bigger therefore hitting the memory problems here:

    grep: memory exhausted
    sed: couldn't re-allocate memory

as both grep and sed try to read/process files line by line which in this case (no separators) leads to loading whole files into memory.

Any idea how to approach this using some another smart one-liner?

share|improve this question
add comment

3 Answers 3

With gawk:

gawk -v 'RS=},{' '{sub(",", "\n", RT); printf "%s", $0 RT}' < file

perl equivalent:

perl -pe 'BEGIN{$/="},{"}; s/\,{$/\n{/' < file

Otherwise, POSIXly:

tr , '\n' < file | awk '{
  if (/^{/ && e) print ""
  printf "%s", $0
  if (/}$/) e=1
  else {e=0; printf ","}}
  END {print ""}'

Pipe those to grep error to see the records with errors, and to paste -sd, - to restore to original format.

share|improve this answer
add comment

You could also do this in Perl:

perl -ne 'BEGIN{$/="},{"} chomp; 
          s/\n$//; s/^{//; s/}$//; 
          print "{$_}\n"; ' k 

This is the same principle as the gawk one that StephaneChazelas suggested, in Perl, $/ is the record separator, so we set that to },{ to read the records correctly and then print them with newlines.

You could easily expand this to do both of the operations you ask for:

perl -i -ne 'BEGIN{$/="},{"}
             chomp; 
             s/\n$//; s/^{//; s/}$//; print "{$_}\n"; 
             print STDERR "{$_}\n" if /error/' file 2> ES01.err.log
share|improve this answer
    
@StephaneChazelas ah, yes, so it does. Sod it then, I'll just leave the perl one. –  terdon Feb 8 at 17:18
    
Note that it adds an extra { to the first record and }\n to the last one. –  Stéphane Chazelas Feb 8 at 17:21
    
@StephaneChazelas this one should work correctly but not really worth it anymore, too complex. Your awk is much better. –  terdon Feb 8 at 18:43
add comment

If you are willing to try a program that is probably not yet installed on your system, try gsar, explained in this answer to the same problem.

gsar is a search and (optionally) replace utility that operates on binary files. It cannot however search with regular expressions.

This command:

gsar '-s},{' '-r}:x0A{' ES01.log > ESnl01.log

replaces the comma between }{ with a newline character, reading from ES01.log and redirecting output to ESnl01.log.

The search (-s) and replacement (-r) strings do not be of the same length.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.