28

I have a document with a lot of empty lines.

How can I remove them when there are 2 or more together.

I tried sed "s/\n\n//" file but it didn't work. No error.

  • 4
    Do I read you correctly if you do not want to remove all blank lines, but only if it is two or more. So not single blank lines? – Runium May 16 '13 at 13:31
  • 1
    And if it's two or more lines shall really all of them be deleted or just all but one? – Hauke Laging May 16 '13 at 13:44
  • You're only asking about replacing empty-lines (^$), not (escaped) double newlines within a line: aaa\n\nbbb? – smci Feb 10 at 23:21
42

Just to remove empty lines:

sed  '/^$/d'

sed is line oriented, so thinking in terms of "2 or more of a particular byte" works except when that byte is a newline. Then you have to think of something that works for the whole line.

| improve this answer | |
  • Of course! +1 for simple elegance. – terdon May 16 '13 at 13:22
  • 2
    sed is capable of handling several lines via its "pattern space" / "hold space" feature. But I feel that's too complicated. ;-) – Hauke Laging May 16 '13 at 13:42
  • 1
    This will not function as desired if the first character of the file is a newline. – Chris Down May 17 '13 at 3:42
  • 2
    To make it work when the first character is a newline (if that's really a requirement), then you can enclose the command with a negative address 1! (match all except line 1), thus: sed '1!{/^$/d'}. – Toby Speight Jan 28 '16 at 19:51
  • 1
    @AaronFranke - yes, but that's a facet of how Linux shells treat '>' redirection. The shell looks at the command line, sees a '>' redirection of stdout to a file, creates that file, and only then runs sed. Creating a file will essentially delete any existing file with the same name. sed '/^&/d' file.txt > otherfile.txt will work. – Bruce Ediger Dec 10 '18 at 19:31
24

No need for sed. grep will do:

grep .

(that's grep, SPC, dot, that is match any line containing at least one character).

There's also:

tr -s '\n'

(squeeze any sequence of newline characters into one).

As noted by Chris, both are not equivalent because removing empty lines (like the first solution above and most other answers focus on here) is not the same as squeezing sequences of newline characters as requested in the case where the first line is empty as it only takes one leading newline character to make the first line empty.

| improve this answer | |
  • 2
    This will not function as desired if the first character of the file is a newline: sprunge.us/FLAJ – Chris Down May 17 '13 at 3:42
9

What do you mean remove? remove duplicate (many blank line to one) or remove all?

If you want to remove duplicate, here is the method using sed:

sed '$!N; /^\(.*\)\n\1$/!P; D'

It simulates uniq command.

The best choice is using awk:

awk NF <filename>
| improve this answer | |
  • The sed part of this works great! Recommending this one as the best answer. – Akito Apr 3 '19 at 8:59
9

If you wanted to keep a single blank line for any given sequence of blank lines you might do:

sed -e '/./b' -e :n -e 'N;s/\n$//;tn'
| improve this answer | |
  • 1
    This is the only answer (besides cat -s) that actually accomplishes exactly what the question asked as I understand it. (And it's better than cat -s because I can use sed -i with it.) – Matthew Jun 5 '18 at 15:25
6

sed is not the best tool for that, since it is line based and treats \n as the end-of-line character this gets complicated. Having seen @Bruce Ediger's answer sed may well be the perfect tool for the job, still, here are some other options:

  1. Perl

    perl -ne 'print if /./' file.txt
    

    or

    perl -pe '$/=""; s/\n+/\n/;' file.txt 
    

    Thanks to @ruakh who made me go and read this:

    $/

    The input record separator, newline by default. This influences Perl's idea of what a "line" is. Works like awk's RS variable, including treating empty lines as a terminator if set to the null string (an empty line cannot contain any spaces or tabs). You may set it to a multi-character string to match a multi-character terminator, or to undef to read through the end of file. Setting it to "\n\n" means something slightly different than setting to "" , if the file contains consecutive empty lines. Setting to "" will treat two or more consecutive empty lines as a single empty line. Setting to "\n\n" will blindly assume that the next input character belongs to the next paragraph, even if it's a newline.

  2. gawk/awk

    awk '$1' file.txt
    

    That will work for the example posted but as @Stephane Chazelas pointed out, it will also delete lines whose first field "looks like" 0. This is more robust:

    awk NF file.txt
    
| improve this answer | |
  • For Perl, perl -pe 's/\n+/\n/ file.txt will do, the input record separator is irrelevant for this use. – vonbrand Jan 28 '16 at 22:09
  • @vonbrand no, perl -pe or perl -ne work line by line. \n+ will never match because it is only applied on a single line. That's why you need to either set $/ or use -0 ti slurp the file whole: perl -0pe 's/\n+/\n/' file. – terdon Jan 29 '16 at 9:48
2

For most of these answers it is first necessary to remove trailing whitespace. Removing doubled up newlines removes all blank lines. (Think about this).

Literally interpreted the OP wants "all blank lines removed from a file if there are any repeated blank lines".

The typical user wants to "remove only duplicated blank lines".

To do this, strip trailing whitepace first, and pipe though cat -s

sed  s/[[:space:]]*$// | cat -s

And yet this will not remove a superflous leading or trailing blank line.

| improve this answer | |
0
  1. Clean up all lines that only have blank spaces and/or tabs by removing the spaces and tabs
  2. Remove all DOUBLE blank lines (i.e. \n\n), while preserving at least one

To do so (with thanks to @cuonglm for part of the code)

sed 's|^[ \t]\+$||' your_filename | sed '$!N; /^\(.*\)\n\1$/!P; D'

Replace your_filename with the actual filename.

| improve this answer | |
-2

Try sed -e 's#\\n\\n#\\n#g' input.file > output.file using / both as your field separator and part of your regex could be the problem.

| improve this answer | |
  • 2
    Just gave this a whirl with one of my files containing double and triple newlines in a sequence. Doesn't work at all for me. – syntaxerror Nov 7 '13 at 19:53
-3

Use this command:

tr -s '\r' '\n'
| improve this answer | |
  • yes, their answer didn't work for me. – meow Jan 28 '16 at 19:03
  • 5
    AFAIK this answer is incorrect. I recommend yo to delete it. – zuazo Jan 28 '16 at 19:10
  • oh, its because my file contains lots of newlines and carriage returns actually. 0x0d0a – meow Jan 28 '16 at 19:11
  • 2
    Actually, the command removes repeated lines with windows end of line. Test with echo -e 'one\r\n\r\n\r\n\rtwo'| tr -s '\r' '\n'. The command tr will translate all \r to \n and then will squeeze all \n to just one. So, it works, not sure what to do with the fact that this apply to windows, not UNIX. – user79743 Jan 28 '16 at 19:32
  • Works for me. Created a file, words in vi with separated by multiple lines. cat file | tr -s '\r' '\n' returned the correct output. – cutrightjm Feb 4 at 1:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.