24

I have a document with a lot of empty lines.

How can I remove them when there are 2 or more together.

I tried sed "s/\n\n//" file but it didn't work. No error.

  • 3
    Do I read you correctly if you do not want to remove all blank lines, but only if it is two or more. So not single blank lines? – Runium May 16 '13 at 13:31
  • 1
    And if it's two or more lines shall really all of them be deleted or just all but one? – Hauke Laging May 16 '13 at 13:44
41

Just to remove empty lines:

sed  '/^$/d'

sed is line oriented, so thinking in terms of "2 or more of a particular byte" works except when that byte is a newline. Then you have to think of something that works for the whole line.

  • Of course! +1 for simple elegance. – terdon May 16 '13 at 13:22
  • 2
    sed is capable of handling several lines via its "pattern space" / "hold space" feature. But I feel that's too complicated. ;-) – Hauke Laging May 16 '13 at 13:42
  • This will not function as desired if the first character of the file is a newline. – Chris Down May 17 '13 at 3:42
  • @ChrisDown - are you certain? I tried it on just such a file, and it seemed to work with (GNU sed) 4.2.2. – Bruce Ediger May 17 '13 at 13:04
  • 1
    To make it work when the first character is a newline (if that's really a requirement), then you can enclose the command with a negative address 1! (match all except line 1), thus: sed '1!{/^$/d'}. – Toby Speight Jan 28 '16 at 19:51
24

No need for sed. grep will do:

grep .

(that's grep, SPC, dot, that is match any line containing at least one character).

There's also:

tr -s '\n'

(squeeze any sequence of newline characters into one).

As noted by Chris, both are not equivalent because removing empty lines (like the first solution above and most other answers focus on here) is not the same as squeezing sequences of newline characters as requested in the case where the first line is empty as it only takes one leading newline character to make the first line empty.

  • 1
    This will not function as desired if the first character of the file is a newline: sprunge.us/FLAJ – Chris Down May 17 '13 at 3:42
7

sed is not the best tool for that, since it is line based and treats \n as the end-of-line character this gets complicated. Having seen @Bruce Ediger's answer sed may well be the perfect tool for the job, still, here are some other options:

  1. Perl

    perl -ne 'print if /./' file.txt
    

    or

    perl -pe '$/=""; s/\n+/\n/;' file.txt 
    

    Thanks to @ruakh who made me go and read this:

    $/

    The input record separator, newline by default. This influences Perl's idea of what a "line" is. Works like awk's RS variable, including treating empty lines as a terminator if set to the null string (an empty line cannot contain any spaces or tabs). You may set it to a multi-character string to match a multi-character terminator, or to undef to read through the end of file. Setting it to "\n\n" means something slightly different than setting to "" , if the file contains consecutive empty lines. Setting to "" will treat two or more consecutive empty lines as a single empty line. Setting to "\n\n" will blindly assume that the next input character belongs to the next paragraph, even if it's a newline.

  2. gawk/awk

    awk '$1' file.txt
    

    That will work for the example posted but as @Stephane Chazelas pointed out, it will also delete lines whose first field "looks like" 0. This is more robust:

    awk NF file.txt
    
  • For Perl, perl -pe 's/\n+/\n/ file.txt will do, the input record separator is irrelevant for this use. – vonbrand Jan 28 '16 at 22:09
  • @vonbrand no, perl -pe or perl -ne work line by line. \n+ will never match because it is only applied on a single line. That's why you need to either set $/ or use -0 ti slurp the file whole: perl -0pe 's/\n+/\n/' file. – terdon Jan 29 '16 at 9:48
4

What do you mean remove? remove duplicate (many blank line to one) or remove all?

If you want to remove duplicate, here is the method using sed:

sed '$!N; /^\(.*\)\n\1$/!P; D'

It simulates uniq command.

The best choice is using awk:

awk NF <filename>
  • The sed part of this works great! Recommending this one as the best answer. – Akito Apr 3 at 8:59
2

For most of these answers it is first necessary to remove trailing whitespace. Removing doubled up newlines removes all blank lines. (Think about this).

Literally interpreted the OP wants "all blank lines removed from a file if there are any repeated blank lines".

The typical user wants to "remove only duplicated blank lines".

To do this, strip trailing whitepace first, and pipe though cat -s

sed  s/[[:space:]]*$// | cat -s

And yet this will not remove a superflous leading or trailing blank line.

  • Downvoted, but this clearly works ? No comment ? – mckenzm Feb 7 '15 at 8:48
  • 1
    I upvoted you for... you know... answering the question. =) I can't believe Bruce Ediger's response got upvoted when it deletes every blank line. If someone asks how to remove duplicated blank lines, I can't imagine any scenario where deleting all blank lines would be an acceptable solution. But whatever. There's a page on the website for sed that covers this, by the way: gnu.org/software/sed/manual/sed.html#cat-_002ds – Todd Walton Oct 30 '17 at 19:39
2

If you wanted to keep a single blank line for any given sequence of blank lines you might do:

sed -e '/./b' -e :n -e 'N;s/\n$//;tn'
  • 1
    This is the only answer (besides cat -s) that actually accomplishes exactly what the question asked as I understand it. (And it's better than cat -s because I can use sed -i with it.) – Matthew Jun 5 '18 at 15:25
-2

Try sed -e 's#\\n\\n#\\n#g' input.file > output.file using / both as your field separator and part of your regex could be the problem.

  • 2
    Just gave this a whirl with one of my files containing double and triple newlines in a sequence. Doesn't work at all for me. – syntaxerror Nov 7 '13 at 19:53
-3

Use this command:

tr -s '\r' '\n'
  • yes, their answer didn't work for me. – meow Jan 28 '16 at 19:03
  • 5
    AFAIK this answer is incorrect. I recommend yo to delete it. – zuazo Jan 28 '16 at 19:10
  • oh, its because my file contains lots of newlines and carriage returns actually. 0x0d0a – meow Jan 28 '16 at 19:11
  • 2
    Actually, the command removes repeated lines with windows end of line. Test with echo -e 'one\r\n\r\n\r\n\rtwo'| tr -s '\r' '\n'. The command tr will translate all \r to \n and then will squeeze all \n to just one. So, it works, not sure what to do with the fact that this apply to windows, not UNIX. – user79743 Jan 28 '16 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.