Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Anyone know of a non-line-based tool to "binary" search/replace strings in a somewhat memory-efficient way? See this question too.

I have a +2GB text file that I would like to process similar to what this appears to do:

sed -e 's/>\n/>/g'

That means, I want to remove all newlines that occur after a >, but not anywhere else, so that rules out tr -d.

This command (that I got from the answer of a similar question) fails with couldn't re-allocate memory :

sed --unbuffered ':a;N;$!ba;s/>\n/>/g'

So, are there any other methods without resorting to C? I hate perl, but am willing to make an exception in this case :-)

I don't know for sure of any character that does not occur in the data, so temporary replacing \n with another character is something I'd like to avoid if possible.

Any good ideas, anyone?

share|improve this question
    
Have you tried option --unbuffered? –  richard Jun 16 at 12:26
    
With or without --unbuffered runs out of memory –  MattBianco Jun 16 at 12:28
    
What does $! do? –  richard Jun 16 at 12:33
    
What is wrong with the first sed command. The second seems to be reading everything into pattern space, I don't know that the $! is though. This I expect will need a LOT of memory. –  richard Jun 16 at 12:35
    
The problem is that sed reads everything as lines, that's why the first command doesn't remove the newlines, since it outputs the text row-by-row again. The second command is just a workaround. I think sed is not the proper tool in this case. –  MattBianco Jun 16 at 12:40

8 Answers 8

up vote 13 down vote accepted

This really is trivial in Perl, you shouldn't hate it!

perl -i.bak -pe 's/>\n/>/' file

Explanation

  • -i : edit the file in place, and create a backup of the original called file.bak. If you don't want a backup, just use perl -i -pe instead.
  • -pe : read the input file line by line and print each line after applying the script given as -e.
  • s/>\n/>/ : the substitution, just like sed.

And here's an awk approach:

awk  '{if(/>$/){printf "%s",$0}else{print}}' file2 
share|improve this answer
3  
+1. awk golf: awk '{ORS=/>$/?"":"\n"}1' –  glenn jackman Jun 16 at 13:03
1  
Why I dislike perl in general is the same reason why I chose this answer (or actually your comment to Gnouc's answer): readability. Using perl -pe with a simple "sed pattern" is way more readable than a complex sed-expression. –  MattBianco Jun 16 at 13:05
3  
@MattBianco fair enough but, just so you know, that has nothing to do with Perl. The lookbehind that Gnouc used is a feature of some regular expression languages (including, but not limited to, PCREs), not Perl's fault at all. Also, after featuring this sed monstrosity ':a;N;$!ba;s/>\n/>/g' in your question, you've waived your right to complain about readability! :P –  terdon Jun 16 at 13:21
    
@glennjackman nice! I was playing with the foo ? bar : baz construct but couldn't get it to work. –  terdon Jun 16 at 13:22
    
@terdon: Yeap, my mistake. Delete it. –  cuonglm Jun 16 at 13:44

A perl solution:

$ perl -pe 's/(?<=>)\n//'

Explaination

  • s/// is used for string substitution.
  • (?<=>) is lookbehind pattern.
  • \n matches newline.

The whole pattern meanings removing all newline that have > before it.

share|improve this answer
2  
care to comment what the parts of the program does? I'm always looking to learn. –  MattBianco Jun 16 at 12:32
2  
Why bother with the lookbehind? Why not just s/>\n/>/? –  terdon Jun 16 at 12:44
1  
or s/>\K\n// would also work –  glenn jackman Jun 16 at 13:00
    
@terdon: Just the first thing I though, remove instead of replace –  cuonglm Jun 16 at 13:13
    
@glennjackman: good point! –  cuonglm Jun 16 at 13:14

You should be able to use sed with the N command, but the trick will be to delete one line from the pattern space each time that you add another (so that the pattern space always contains only 2 consecutive lines, instead of trying to read in the whole file) - try

sed ':a;$!N;s/>\n/>/;P;D;ba'

EDIT: after re-reading Peteris Krumins' Famous Sed One-Liners Explained I believe a better sed solution would be

sed -e :a -e '/>$/N; s/\n//; ta'

which only appends the following line in the case that it's already made a > match at the end, and should conditionally loop back to handle the case of consecutive matching lines (it is Krumin's 39. Append a line to the next if it ends with a backslash "\" exactly except for the substitution of > for \ as the join character, and the fact that the join character is retained in the output).

share|improve this answer
1  
That doesn't work if 2 consecutive lines end in > (that's also GNU specific) –  Stéphane Chazelas Jun 16 at 12:51

How about this:

sed ':loop
  />$/ { N
    s/\n//
    b loop
  }' file

For GNU sed, you can also try adding the -u (--unbuffered) option as per the question. GNU sed is also happy with this as a simple one-liner:

sed ':loop />$/ { N; s/\n//; b loop }' file
share|improve this answer
    
That doesn't remove the last \n if the file ends in >\n, but that's probably preferable anyway. –  Stéphane Chazelas Jun 16 at 12:49
    
@StéphaneChazelas, why does the closing } need to be in a separate expression? will this not work as a multiline expression? –  Graeme Jun 16 at 12:56
1  
That will work in POSIX seds with b loop\n} or -e 'b loop' -e '}' but not as b loop;} and certainly not as b loop} because } and ; are valid in label names (though nobody in their right mind would use it. And that means GNU sed is not POSIX conformant) and the } command needs to be separated from the b command. –  Stéphane Chazelas Jun 16 at 13:00
    
@StéphaneChazelas, GNU sed is happy with all of the above even with --posix! The standard also has the following for brace expressions - The list of sed functions shall be surrounded by braces and separated by <newline>s. Does this not mean that semicolons should only be used outside of braces? –  Graeme Jun 16 at 13:15
    
@mikeserv, the loop is needed to handle consecutive lines ending in >. The original never had one, this was pointed out by Stéphane. –  Graeme Jun 16 at 13:58

There are a lot of ways to do this, and most here are really good, but I think this one's my favorite:

tr '>\n' '\n>' | sed 's/^>*//;H;/./!d;x;y/\n>/>\n/'

Or even:

tr '>\n' '\n>' | sed 's/^>*//' | tr '\n>' '>\n'
share|improve this answer
    
I can’t get your first answer to work at all. While I admire the elegance of the second one, I believe that you need to remove the *. The way it is now, it will delete any blank lines following a line that ends with a >. … Hmm. Looking back at the question, I see that it’s a little ambiguous. The question says, “I want to remove all newlines that occur after a >, …” I interpret that to mean that >\n\n\n\n\nfoo should be changed to \n\n\n\nfoo, but I suppose foo might be the desired output. –  Scott Jun 17 at 18:55
    
@Scott - I tested with variations on the following: printf '>\n>\n\n>>\n>\n>>>\n>\nf\n\nff\n>\n' | tr '>\n' '\n>' | sed 's/^>*//;H;/./!d;x;y/\n>/>\n/' - that results in >>>>>>>>>>f\n\nff\n\n for me with the first answer. I am curious though what you're doing to break it though, because I'd like to fix it. As to the second point - I don't agree that it is ambiguous. The OP does not ask to remove all > preceding a \newline, but instead to remove all \newlines following a >. –  mikeserv Jun 17 at 19:01
1  
Yes, but a valid interpretation is that, in >\n\n\n\n\n, only the first newline is after a >; all the others are following other newlines. Note that the OP’s “this is what I want, if only it worked” suggestion was sed -e 's/>\n/>/g', not sed -e 's/>\n*/>/g'. –  Scott Jun 17 at 19:09
    
@Scott - the suggestion did not work and never could. I don't believe that the code suggestion of someone who does not fully understand the code can be considered as valid an interpreting point as the plain language that person also uses. And besides, the output - if it actually worked - of s/>\n/>/ on >\n\n\n\n\n would still be something that s/>\n/>/ would edit. –  mikeserv Jun 17 at 19:11

sed doesn't provide a way to emit output without a final newline. Your approach using N fundamentally works, but stores incomplete lines in memory, and thus can fail if the lines become too long (sed implentations aren't typically designed to handle extremely long lines).

You can use awk instead.

awk '{if (/<$/) printf "%s", $0; else print}'

An alternative approach is to use tr to swap the newline character with a “boring”, frequently-occurring character. Space might work here — pick a character that tends to appear on every line or at least a large proportion of lines in your data.

tr ' \n' '\n ' | sed 's/> />/g' | tr '\n ' ' \n'
share|improve this answer
    
Both methods are already demonstrated here to better effect in other answers. And his approach with sed does not work without a 2.5gigabyte buffer. –  mikeserv Jun 17 at 2:29
    
Did anybody mention awk? Oh, I missed it, I'd only noticed perl in terdon's answer for some reason. Nobody mentioned the tr approach — mikeserv, you posted a different (valid, but less generic) approach that happens to also use tr. –  Gilles Jun 17 at 7:55
    
valid, but less generic sounds to me like youve just called it a working, targeted solution. i think its hard to argue that such a thing isnt useful which is odd because it has 0 upvotes. The biggest difference i can see between my own solution and your more generic offering, is that mine specifically solves a problem, whereas yours might generally. That might make it worthwhile - and i may even reverse my vote - but theres also the pesky matter of the 7 hours between them and the recurring theme of your answers mimicking others. Can you explain this? –  mikeserv Jun 17 at 16:48

I ended up using gsar as described in this answer like this:

gsar -F '-s>:x0A' '-r>'
share|improve this answer

what about using ed?

ed -s test.txt <<< $'/fruits/s/apple/banana/g\nw'

(via http://wiki.bash-hackers.org/howto/edit-ed)

share|improve this answer
    
edited, there is no dependency on website anymore –  andrej Nov 19 at 9:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.