Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm developing on a max osx 10.8 and have no preferences to which tool to use probably sed but it doesn't matter.

Update after comment: The input file uses windows new lines if that's complicates things I can pre process them to change the new lines to linux style.






And I would like an output file like this



share|improve this question
Please format the posted data in a way we can use it for testing our solutions to be proposed. – manatwork Aug 12 '13 at 16:32
A new format is posted, let me know if this suffice. – Farmor Aug 12 '13 at 16:52
up vote 1 down vote accepted

Or a perl variant:

perl -e 'undef $/; $_=<>; s/^\n{2,}//mg; print' < infile

unset (undef) the field separator ($/). This makes $_=<> assign the contents of STDIN into $_. Then search and replace 2 or more newlines with nothing (s/^\n{2,}//mg;), and print the output (these operations are done on $_). The m option on the search s/// above allows newline (multiple line) matching and g applies the pattern for all occurances in the string (and not just the first).

If it is to only work on 2 and 3 new lines (and not effect 4 or 5 etc) then use s/^\n{2,3}//mg;

For DOS format file:

perl -e 'undef $/; $_=<>; s/^(\r\n){2,}//mg; print' < infile
share|improve this answer
Thanks, worked really good. It had a shockingly fast execution time to. The only small thing, and it doesn't matter for me, was that it didn't remove 2 new lines at the beginning of the file. – Farmor Aug 12 '13 at 22:01
Odd I'd tested for that in my tests :) May be those lines contain whitespace (tabs, spaces?) Glad it wasn't an issue! – Drav Sloan Aug 12 '13 at 23:53
You were correct it removes the first line also if formatted with utf8. Sadly the downloaded file was encoded with utf8 with BOM. Removing the BOM and the script worked fine also for the first two lines. – Farmor Aug 13 '13 at 0:33
Same as perl -0777 -pe's/^\n{2,}//mg'. It should be noted that since it slurps the whole file in memory, it's not adapted to huge files. – Stéphane Chazelas Aug 13 '13 at 6:07
I don't think that variant works, because doesn't -p split the STDIN across newlines (and so you can't do multiline matching?) – Drav Sloan Aug 13 '13 at 11:04
awk -vRS='\r\n' -vORS='\r\n' '
  /./||!++n{if (n==1) print "";print;n=0}
  END{if (n==1) print ""}'
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.