I have files that were generated by a program that did not put newlines at the end of records. I want to put newlines between the records, and I can do so with a simple sed script:

sed -e 's/}{/}\n{/g'

The problem is that the input files are multiple gigabytes in size, and therefore the input lines to sed are multiple GBs in length. sed tries to hold a line in memory, which doesn't work in this case. I tried the --unbuffered option, but that just seemed to make it slower and did not allow it to finish correctly.

  • Would it be possible to upload an example input file somewhere for us to try some ideas? – mkc Mar 2 '15 at 15:35
  • 3
    Maybe you could first use tr to translate } into \n and then use sed to add a } at the end of each line? Like this : tr '}' '\n' < your_file.txt| sed 's/$/}/' – user43791 Mar 2 '15 at 15:36
  • Does adding a newline at the end of the file help at all? Like: printf "\n" >> file – nanny Mar 2 '15 at 15:36
  • 1
    @Ketan, I assume writing a file with 78 garbage characters following by }{ repeated until it's several gigabytes long would suffice. – nanny Mar 2 '15 at 15:38
  • @nanny - good point - but where do you get 78? If the records are already blocked then dd if=file cbs=80 conv=unblock would do it - but it's rarely that simple. – mikeserv Mar 2 '15 at 19:38

You can use another tool that lets you set the input record separator. For example

  • Perl

    perl -pe 'BEGIN{ $/="}{" } s/}{/}\n{/g' file

    The special variable $/ is the input record separator. Setting it to }{ defines lines as ending in }{. That way you can achieve what you want without reading the entire thing into memory.

  • mawk or gawk

    awk -v RS="}{" -vORS= 'NR > 1 {print "}\n{"}; {print}' file 

    This is the same idea. RS="}{" sets the record separator to }{ and then you print }, a newline, { (except for the first record) and the current record.

| improve this answer | |

Perl to the rescue:

perl -i~ -e ' $/ = \1024;
              while (<>) {
                  print "\n" if $closing and /^{/;
                  undef $closing;
                  $closing = 1 if /}$/;
              } ' input1 input2

Setting $/ to \1024 will read the file in chunks of 1024 bytes. The $closing variable handles the case when a chunk ends in } and the next one starts with {.

| improve this answer | |
  • 1
    +1, probably the best solution ; the other perl/awk solutions work fine too but what if the first record separator occurs after about 17GB worth of chars ? – don_crissti Mar 2 '15 at 21:17

You should do:

{ <infile tr \} \\n;echo {; } | paste -d'}\n' - /dev/null >outfile

It's probably the most efficient solution.

That puts a {} to protect any possible trailing data. With one more tr process you can swap that around and do a blank line at the head of the first { field. Like...

tr {} '}\n'| paste -d{\\0 /dev/null - | tr {}\\n \\n{}

So the first, with don's example data, does:

printf '{one}{two}{three}{four}' |
{ tr \} \\n; echo {; }           |
paste -d'}\n' - /dev/null

...and the second one does...

printf '{one}{two}{three}{four}'      |
tr {} '}\n'| paste -d{\\0 /dev/null - |
tr {}\\n \\n{}
#leading blank

There is no trailing newline for the second example - though there is one for the first.

| improve this answer | |

A Binary sed-like utility called bbe

I find it easiest to stay with sed-like syntax in this case.

I much prefer using the bbe utility (available via your {uni,linu}x's package installation, e.q. apt-get). Or here if you're one of the git crowd, though I haven't personally vetted that particular link.

1. It supports the s/before/after/ idiom

It's a "Binary Block Editor", which supports sed-like (among other) operations. This includes the super common s/before/after/ substitution idiom that you need. Note, because there are no lines per se from bbe's point of view, there is no "global g" at the end of the command.

As a quick test (note the required -e):

$ echo hello | bbe -e 's/l/(replaced)/'



2. In your specific case of }{ to }\n{ conversion

So if we had a massive file filled with a million numbers in (say) the format {1}{2}{3}...{1000000} with no carriage returns, we could exchange the }{ with }\n{ easily, and have all the numbers one per line.

This would be with this bbe command:

bbe -e 's/}{/}\n{/'

As tested in this zsh loop, which we grab just the tail of:

$ for ((num=0; num<1000000; num++)) do; echo -n "{$num}"; done | bbe -e 's/}{/}\n{/' | tail

Which would produce this:


(without a trailing carriage return of course.)

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.