Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am currently trying to remove all newlines that are not preceded by a closing parenthesis, so I came up with this expression:

sed -r -i -e ":a;N;$!ba;s/([^\)])\n/\1/g;d" reallyBigFile.log

It does the job on smaller files, but on this large file I am using (3GB), it works for a while then returns with an out of memory error:

sed: Couldn't re-allocate memory

Is there any way I could do this job without running into this issue. Using sed itself is not mandatory, I just want to get it done.

share|improve this question
    
Use split to create more files out of the big one, then recompose it with cat. –  schaiba Feb 1 '13 at 13:31
    
There are many instances where i dont have a ending parentheses, if I split, probably I will not solve all of them. –  kurast Feb 1 '13 at 13:45
add comment

2 Answers

up vote 3 down vote accepted

Your first three commands are the culprit:

:a
N
$!ba

This reads the entire file into memory at once. The following script should only keep one segment in memory at a time:

% cat test.sed
#!/usr/bin/sed -nf

# Append this line to the hold space. 
# To avoid an extra newline at the start, replace instead of append.
1h
1!H

# If we find a paren at the end...
/)$/{
    # Bring the hold space into the pattern space
    g
    # Remove the newlines
    s/\n//g 
    # Print what we have
    p
    # Delete the hold space
    s/.*//
    h
}
% cat test.in
a
b
c()
d()
e
fghi
j()
% ./test.sed test.in
abc()
d()
efghij()

This awk solution will print each line as it comes, so it will only have a single line in memory at a time:

% awk '/)$/{print;nl=1;next}{printf "%s",$0;nl=0}END{if(!nl)print ""}' test.in
abc()
d()
efghij()
share|improve this answer
    
Great answer (with included sed tutorial)! +1 –  vonbrand Feb 1 '13 at 17:24
    
It just prints abc() and stops? –  kurast Feb 1 '13 at 17:31
    
I have just checked and it does not stop. Thank you –  kurast Feb 1 '13 at 17:39
add comment

For completeness, a Perl solution: perl -p -e '/)$/ || chomp'

For symmetry: -p wraps your script in a loop reading and printing line by line; the -e expression/script matches ) at the end of the line, if it doesn't match (match is false) it goes on to chomp, which removes the newline at the end.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.