Replace string containing newline in huge file

Question

Anyone know of a non-line-based tool to "binary" search/replace strings in a somewhat memory-efficient way? See this question too.

I have a +2GB text file that I would like to process similar to what this appears to do:

sed -e 's/>\n/>/g'

That means, I want to remove all newlines that occur after a >, but not anywhere else, so that rules out tr -d.

This command (that I got from the answer of a similar question) fails with couldn't re-allocate memory :

sed --unbuffered ':a;N;$!ba;s/>\n/>/g'

So, are there any other methods without resorting to C? I hate perl, but am willing to make an exception in this case :-)

I don't know for sure of any character that does not occur in the data, so temporary replacing \n with another character is something I'd like to avoid if possible.

Any good ideas, anyone?

Answer 1

This really is trivial in Perl, you shouldn't hate it!

perl -i.bak -pe 's/>\n/>/' file

Explanation

• -i : edit the file in place, and create a backup of the original called file.bak. If you don't want a backup, just use perl -i -pe instead.
• -pe : read the input file line by line and print each line after applying the script given as -e.
• s/>\n/>/ : the substitution, just like sed.

And here's an awk approach:

awk  '{if(/>$/){printf "%s",$0}else{print}}' file2 

Answer 2

A perl solution:

$ perl -pe 's/(?<=>)\n//'

Explaination

• s/// is used for string substitution.
• (?<=>) is lookbehind pattern.
• \n matches newline.

The whole pattern meanings removing all newline that have > before it.

Answer 3

How about this:

sed ':loop
  />$/ { N
    s/\n//
    b loop
  }' file

For GNU sed, you can also try adding the -u (--unbuffered) option as per the question. GNU sed is also happy with this as a simple one-liner:

sed ':loop />$/ { N; s/\n//; b loop }' file

Answer 4

You should be able to use sed with the N command, but the trick will be to delete one line from the pattern space each time that you add another (so that the pattern space always contains only 2 consecutive lines, instead of trying to read in the whole file) - try

sed ':a;$!N;s/>\n/>/;P;D;ba'

---

EDIT: after re-reading Peteris Krumins' Famous Sed One-Liners Explained I believe a better sed solution would be

sed -e :a -e '/>$/N; s/\n//; ta'

which only appends the following line in the case that it's already made a > match at the end, and should conditionally loop back to handle the case of consecutive matching lines (it is Krumin's 39. Append a line to the next if it ends with a backslash "\" exactly except for the substitution of > for \ as the join character, and the fact that the join character is retained in the output).

Answer 5

sed doesn't provide a way to emit output without a final newline. Your approach using N fundamentally works, but stores incomplete lines in memory, and thus can fail if the lines become too long (sed implentations aren't typically designed to handle extremely long lines).

You can use awk instead.

awk '{if (/<$/) printf "%s", $0; else print}'

An alternative approach is to use tr to swap the newline character with a “boring”, frequently-occurring character. Space might work here — pick a character that tends to appear on every line or at least a large proportion of lines in your data.

tr ' \n' '\n ' | sed 's/> />/g' | tr '\n ' ' \n'

Answer 6

what about using ed?

ed -s test.txt <<< $'/fruits/s/apple/banana/g\nw'

(via http://wiki.bash-hackers.org/howto/edit-ed)

Answer 7

I ended up using gsar as described in this answer like this:

gsar -F '-s>:x0A' '-r>'

Answer 8

There are a lot of ways to do this, and most here are really good, but I think this one's my favorite:

tr '>\n' '\n>' | sed 's/^>*//;H;/./!d;x;y/\n>/>\n/'

Or even:

tr '>\n' '\n>' | sed 's/^>*//' | tr '\n>' '>\n'