16

I have some text-files I use to take notes in - just plain text, usually just using cat >> file. Occasionally I use a blank line or two (just return - the new-line character) to specify a new subject/line of thought. At the end of each session, before closing the file with Ctrl+D, I typically add lots (5-10) blank lines (return-key) just to separate the sessions.

This is obviously not very clever, but it works for me for this purpose. I do however end-up with lots and lots of unnecessary blank lines, so I'm looking for a way to remove (most of) the extra lines. Is there a Linux-command (cut, paste, grep, ...?) that could be used directly with a few options? Alternatively, does anybody have an idea for a sed, awk or perl (well in any scripting-language really, though I'd prefer sed or awk) script that would do what I want? Writing something in C++ (which I actually could do myself), just seems like overkill.

Case #1: What I need is a script/command that would remove more than two (3 or more) consecutive blank lines, and replace them with just two blank lines. Though it would be nice if it also could be tweaked to remove more than one line (2 or more) and/or replace multiple blank lines with just one blank line.

Case #2: I could also use a script/command that would remove a single blank line between two lines of text, but leave multiple blank lines as is (though removing one of the blank lines would also be acceptable).

2

8 Answers 8

16

Case 1:

awk '!NF {if (++n <= 2) print; next}; {n=0;print}'

Case 2:

awk '!NF {s = s $0 "\n"; n++; next}
     {if (n>1) printf "%s", s; n=0; s=""; print}
     END {if (n>1) printf "%s", s}'
1
  • Since this use case is repeated frequently, I would suggest creating a script. Oct 10, 2013 at 23:43
15

You can use uniq to collapse multiple instance of blank lines into one blank line, but it will also collapse lines which contain text if they are the same and below each other.

0
6

Case 1:

perl -i -ane '$n=(@F==0) ? $n+1 : 0; print if $n<=2'

Case 2:

perl -i -ane '$n=(@F==0) ? $n+1 : 0; print $n==2 ? "\n$_" : $n==1 ? "" : $_ '
1
  • +1 perl ftw! Awk is (probably) canonical for this, but (DRY) compels me to write scripts for use-cases that are repeated like this. Oct 10, 2013 at 23:42
5

You can address Case #1 like this with GNU sed:

sed -r ':a; /^\s*$/ {N;ba}; s/( *\n *){2,}/\n\n/'

That is, collect empty lines in pattern space, and if there are more than three or more lines, reduce it to two lines.

To join single-spaced lines, as in Case #2, you can do it like this:

sed -r '/^ *\S/!b; N; /\n *$/!b; N; /\S *$/!b; s/\n *\n/\n/'

Or in commented form:

sed -r '
  /^ *\S/!b        # non-empty line
  N                # 
  /\n *$/!b        # followed by empty line
  N                # 
  /\S *$/!b        # non-empty line
  s/\n *\n/\n/     # remove the empty line
'
1

Following Anthon's suggestion to use "uniq"...

Remove leading, trailing and duplicate blank lines.

# Get large random string.
rand_str=; while [[ ${#rand_str} -lt 40 ]]; do rand_str=$rand_str$RANDOM; done

# Add extra lines at beginning and end of stdin.
(echo $rand_str; cat; echo $rand_str) |

# Convert empty lines to random strings.
sed "s/^$/$rand_str/" |

# Remove duplicate lines.
uniq |

# Remove first and last line.
sed '1d;$d' |

# Convert random strings to empty lines.
sed "s/$rand_str//"

In one long line:

(rand_str=; while [[ ${#rand_str} -lt 40 ]]; do rand_str=$rand_str$RANDOM; done; (echo $rand_str; cat; echo $rand_str) | sed "s/^$/$rand_str/" | uniq | sed '1d;$d' | sed "s/$rand_str//")

Or just use "cat -s".

I switched from parenthesis to curly braces in order to remain in the current shell context which I assume is more efficient. Note that curly braces require semicolon after last command and need a space for separation.

# Add extra blank lines at beginning and end.
# These will be removed in final step.
{ echo; cat; echo; } |

# Replace multiple blank lines with a single blank line.
cat -s |

# Remove first and last line.
sed '1d;$d'

In a single line.

{ { echo; cat; echo; } | cat -s | sed '1d;$d'; }
1

This solution takes care also of the last blank lines in the file:

sed -r -n '
  /^ *$/!{p;b}  # non-blank line - print and next cycle
  h             # blank line - save it in hold space
  :loop
  $b end        # last line - go to end
  n             # read next line in pattern space
  /^ *$/b loop  # blank line - loop to next one
  :end          # pattern space has non-blank line or last blank line
  /^ *$/{p;b}   # last blank line: print and exit
  H;x;p         # non-blank line: print hold + pattern space and next cycle
'
1

Here is a simple way which removes multiple blank lines and replaced with single blank line; using sed, uniq, and tee commands.

$ cat myfile | uniq | tee myfile
1
  • Slick and it works (assuming no dup lines, like in code and config). Leaves some dups but only if the lines have whitespace chars. Jan 16, 2022 at 19:50
0

The posted solutions looked a little bit cryptic to me. Here is the solution in Python 3.6:

#!/usr/bin/env python3

from pathlib import Path                                                                                                                                                              
import sys                                                                                                                                                                            
import fileinput                                                                                                                                                                      


def remove_multiple_blank_lines_from_file(path, strip_right=True): 
    non_blank_lines_out_of_two_last_lines = [True, True] 
    for line in fileinput.input(str(path), inplace=True): 
        non_blank_lines_out_of_two_last_lines.pop(0) 
        non_blank_lines_out_of_two_last_lines.append(bool(line.strip())) 
        if sum(non_blank_lines_out_of_two_last_lines) > 0: 
            line_to_write = line.rstrip() + '\n' if strip_right else line 
            sys.stdout.write(line_to_write)


def remove_multiple_blank_lines_by_glob(rglob='*', path=Path('.'), strip_right=True): 
    for p in path.rglob(rglob): 
        if p.is_file(): 
            try:
                remove_multiple_blank_lines_from_file(p, strip_right=strip_right)
            except Exception as e:
                print(f"File '{p}' was not processed due the error: {e}")


if __name__ == '__main__':
    remove_multiple_blank_lines_by_glob(sys.argv[1], Path(sys.argv[2]), next(iter(sys.argv[3:]), None) == '--strip-right')

You can call the functions from an interpreter or run it from the shell like:

$ ./remove_multiple_lines.py '*' /tmp/ --strip-right
1
  • This is as cryptic to those who find the syntax of python horrid as the others are cryptic to those who don't know those commands. Also some of those are commented and this one is not. This is also much longer. Not saying it's bad - just saying that it's not any better wrt being cryptic.
    – Pryftan
    Nov 2, 2023 at 12:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .