Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have some text-files I use to take notes in - just plain text, usually just using cat >> file. Occasionally I use a blank line or two (just return - the new-line character) to specify a new subject/line of thought. At the end of each sessions - before closing the file with ctrl-D - I typically add lots (5-10) blank lines (return-key) just to separate the sessions.

This is obviously not very cleaver, but it work for me for this purpose. I do however end-up with lots and lots of unnecessary blank lines, so I'm looking for a way to remove (most of) the extra lines. Is there a Linux-command (cut, paste, grep, ...?) that could be used directly with a few options? Alternatively, does anybody have an idea for a sed, awk or perl (well in any scripting-language really, though I'd prefer sed or awk) script that would do what I want? Writing something in C++ (which I actually could do myself), just seems like overkill...

Case #1: What I need is a script/command that would remove more than two (3 or more) consecutive blank lines, and replace them with just two blank lines. Though it would be nice if it also could be tweaked to remove more than one line (2 or more) and/or replace multiple blank lines with just one blank line.

Case #2: I could also use a script/command that would remove a single blank line between two lines of text, but leave multiple blank lines as is (though removing one of the blank lines would also be acceptable).

PS: Yes, I absolutely ought to learn advanced(?) sed and awk scripting, as well as perl... it's long overdue. But as I promised myself to do just that after reading a very good Linux-book back in 1995, I'm not that hopeful I will.

share|improve this question
possible duplicate of Replacing Multiple blank lines with a single blank line in vim / sed – l0b0 Apr 17 '13 at 10:45
@l0b0, that's a completely different question altogether (the other one was a vim one, and was to replace blank lines with one blank line). – Stéphane Chazelas Apr 17 '13 at 11:41
up vote 12 down vote accepted

Case 1:

awk '!NF {if (++n <= 2) print; next}; {n=0;print}'

Case 2:

awk '!NF {s = s $0 "\n"; n++; next}
     {if (n>1) printf "%s", s; n=0; s=""; print}
     END {if (n>1) printf "%s", s}'
share|improve this answer
+1 for awk instead of sed – Rob Apr 17 '13 at 17:22
Since this use case is repeated frequently, I would suggest creating a script. – ChuckCottrill Oct 10 '13 at 23:43

You can use uniq to collapse multiple instance of blank lines into one blank line, but it will also collapse lines which contain text if they are the same and below each other.

share|improve this answer
+1 for using uniq in a way that's never occurred to me. – Baard Kopperud Apr 17 '13 at 11:00

Case 1:

perl -i -ane '$n=(@F==0) ? $n+1 : 0; print if $n<=2'

Case 2:

perl -i -ane '$n=(@F==0) ? $n+1 : 0; print $n==2 ? "\n$_" : $n==1 ? "" : $_ '
share|improve this answer
+1 perl ftw! Awk is (probably) canonical for this, but (DRY) compels me to write scripts for use-cases that are repeated like this. – ChuckCottrill Oct 10 '13 at 23:42

You can address Case #1 like this with GNU sed:

sed -r ':a; /^\s*$/ {N;ba}; s/( *\n *){2,}/\n\n/'

That is, collect empty lines in pattern space, and if there are more than three or more lines, reduce it to two lines.

To join single-spaced lines, as in Case #2, you can do it like this:

sed -r '/^ *\S/!b; N; /\n *$/!b; N; /\S *$/!b; s/\n *\n/\n/'

Or in commented form:

sed -r '
  /^ *\S/!b        # non-empty line
  N                # 
  /\n *$/!b        # followed by empty line
  N                # 
  /\S *$/!b        # non-empty line
  s/\n *\n/\n/     # remove the empty line
share|improve this answer

Following Anthon's suggestion to use "uniq"...

Remove leading, trailing and duplicate blank lines.

# Get large random string.
rand_str=; while [[ ${#rand_str} -lt 40 ]]; do rand_str=$rand_str$RANDOM; done

# Add extra lines at beginning and end of stdin.
(echo $rand_str; cat; echo $rand_str) |

# Convert empty lines to random strings.
sed "s/^$/$rand_str/" |

# Remove duplicate lines.
uniq |

# Remove first and last line.
sed '1d;$d' |

# Convert random strings to empty lines.
sed "s/$rand_str//"

In one long line:

(rand_str=; while [[ ${#rand_str} -lt 40 ]]; do rand_str=$rand_str$RANDOM; done; (echo $rand_str; cat; echo $rand_str) | sed "s/^$/$rand_str/" | uniq | sed '1d;$d' | sed "s/$rand_str//")

Or just use "cat -s".

I switched from parenthesis to curly braces in order to remain in the current shell context which I assume is more efficient. Note that curly braces require semicolon after last command and need a space for separation.

# Add extra blank lines at beginning and end.
# These will be removed in final step.
{ echo; cat; echo; } |

# Replace multiple blank lines with a single blank line.
cat -s |

# Remove first and last line.
sed '1d;$d'

In a single line.

{ { echo; cat; echo; } | cat -s | sed '1d;$d'; }
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.