The 5GB files I have are streams of data rows formed:


so actually could say that there are lines {}, and even line separators, but coming as a three char sequence: },{

I want to do two things:

  1. print "lines" that have string "error" in it:

    grep -o -P {[^{}]+?error.+?} ES01.log > ES01.err.log
  2. make the file more "friendly" by explicitly producing files with new line separators

    <ES01.log sed -e 's/},{/}\n{/g' > ESnl01.log

While the above works for relatively small files (up to ~100MB), my files are unfortunately a lot bigger therefore hitting the memory problems here:

    grep: memory exhausted
    sed: couldn't re-allocate memory

as both grep and sed try to read/process files line by line which in this case (no separators) leads to loading whole files into memory.

Any idea how to approach this using some another smart one-liner?


With gawk:

gawk -v 'RS=},{' '{sub(",", "\n", RT); printf "%s", $0 RT}' < file

perl equivalent:

perl -pe 'BEGIN{$/="},{"}; s/\,{$/\n{/' < file

Otherwise, POSIXly:

tr , '\n' < file | awk '{
  if (/^{/ && e) print ""
  printf "%s", $0
  if (/}$/) e=1
  else {e=0; printf ","}}
  END {print ""}'

Pipe those to grep error to see the records with errors, and to paste -sd, - to restore to original format.


You could also do this in Perl:

perl -ne 'BEGIN{$/="},{"} chomp; 
          s/\n$//; s/^{//; s/}$//; 
          print "{$_}\n"; ' k 

This is the same principle as the gawk one that StephaneChazelas suggested, in Perl, $/ is the record separator, so we set that to },{ to read the records correctly and then print them with newlines.

You could easily expand this to do both of the operations you ask for:

perl -i -ne 'BEGIN{$/="},{"}
             s/\n$//; s/^{//; s/}$//; print "{$_}\n"; 
             print STDERR "{$_}\n" if /error/' file 2> ES01.err.log
  • @StephaneChazelas ah, yes, so it does. Sod it then, I'll just leave the perl one. – terdon Feb 8 '14 at 17:18
  • Note that it adds an extra { to the first record and }\n to the last one. – Stéphane Chazelas Feb 8 '14 at 17:21
  • @StephaneChazelas this one should work correctly but not really worth it anymore, too complex. Your awk is much better. – terdon Feb 8 '14 at 18:43

If you are willing to try a program that is probably not yet installed on your system, try gsar, explained in this answer to the same problem.

gsar is a search and (optionally) replace utility that operates on binary files. It cannot however search with regular expressions.

This command:

gsar '-s},{' '-r}:x0A{' ES01.log > ESnl01.log

replaces the comma between }{ with a newline character, reading from ES01.log and redirecting output to ESnl01.log.

The search (-s) and replacement (-r) strings do not be of the same length.


You could do this simply through Perl using regex.

perl -pe 's/(?<=}),(?=\{)/\n/g' file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.