How to remove visually empty lines from a text file?

Question

I have a bunch of text files, some of which contain lines that are empty, i.e. only consist only of a newline, or possible spaces followed by a newline. I locate the files using a find command.

• Example file

  #Title 1
  12345678 1234
  
  #Title 2
  12345678 1234
  12345678 1234
  
  
  
  
  
  
  
  

• Expected output

  #Title 1
  12345678 1234
  #Title 2
  12345678 1234
  12345678 1234
  

I want to remove all such empty lines. I tried it with the following command on Debian Linux Stretch:

cat "/path/to/file" | sed '/^\s*$/d' | sponge "/path/to/file";

Some of the files had for example 4 or more trailing blank lines, but the above command only removed all but one of the trailing blank lines.

How could I remove this last trailing blank line? As mentioned: should there be any blank lines further up in the file then these should be removed also.

I am trying to get some consistency between the files as the files are stored in an array of sorts within a BASH variable. The files are then looped over and all the blank lines and trailing blank lines are removed, while some of the files already don't have blank lines or any trailing blank line.

Answer 1

If I understand your question correctly, you want to remove (truly or visually) empty lines from a text file. This can be done easily using awk.

For a single file, you could call

awk 'NF' /path/to/file

This will only print files that have at least one "non-blank" character on the line. The idea behind this is that awk by default splits input lines into "fields" at "whitespace", i.e. contiguous runs of space and tab characters. However, if a line only consists of such characters, the number of fields, stored internally in the automatic variable NF, will be recognized as "zero". The (rather short) program above imposes the condition that NF must be non-zero in order for it to print the current line. This effectively removes truly or "visually" empty lines.

Since awk by default will not perform inline editing, you may either have to resort to redirecting the output to a temporary file and then renaming, or using a sufficiently recent implementation that understands the -i inplace extension:

awk -i inplace 'NF' /path/to/file

Answer 2

Here's another portable approach, which is to include only lines that contain something other than whitespace:

grep '[^[:space:]]' file

You can use the same approach with other commands too:

sed -n '/[^[:space:]]/p' file

Writing to the same file as the source is a fairly standard procedure. Some commands use -i (or an equivalent) to indicate in place editing, but in practice they actually write to a temporary file and then overwrite the original file with the temporary:

some_command file >file.tmp && mv -f file.tmp file
rm -f file.tmp

That works well provided that file didn't have hard-links from other places. To cater for such situations you need a double copy:

some_command file >file.tmp && cat file.tmp >file
rm -f file.tmp

Answer 3

I can unfortunately only reproduce your issue on macOS, where sed understands \s as s. The pattern ^\s*$ would therefore match any line consisting of zero or more s characters. This includes empty lines, but not lines that contain only space-like characters.

---

A portable way of removing lines that are empty or that only contain spaces or tabs is

grep -v -x '[[:blank:]]*' file

This uses grep to extract only lines that do not match [[:blank:]]*. The [[:blank:]]* pattern matches zero or more spaces or tab characters. If you want to match a larger set of space-like characters (including things like carriage returns and vertical tabs), use [[:space:]]* instead. The -x option to grep forces the pattern to match complete lines (as if you had anchored the expression with both ^ and $).

Answer 4

You can just replace the \s with [[:space:]]. In addition, most sed versions have a -i flag which means edit the file in place. So this command should work:

sed -i '/^[[:space:]]*$/d' /path/to/file

Answer 5

You could use:

grep '[[:graph:]]'

Which would report the lines containing at least one graphical character, so excluding lines that are empty or consist only of whitespace character, control characters, unknown/undefined/invalid characters.

Answer 6

Testing that on a system with GNU tools; here, the % is the shell prompt, and the $ the end-of-line marker added by cat -A:

% printf 'foo\n   \nbar\n   \n    \n' > file.txt
% cat -A file.txt
foo$
   $
bar$
   $
    $
% cat file.txt | sed '/^\s*$/d' | sponge file.txt
% cat file.txt  
foo
bar
%

There are no empty lines at the end of the resulting file.

The fact that some editors allow moving the cursor under the last line to make it easier to add a new line at the end is unrelated to how the apparently-empty lines get removed by that sed command.

(Instead of the pipe with sponge, you could just use sed -i '/^\s*$/d' file.txt, but you probably should use [[:space:]]* rather than \s* is that one is more widely supported.)

Answer 7

Using Raku (formerly known as Perl_6)

To remove blank (i.e. non-character containing) lines:

raku -ne '.put if .chars;' 

To remove blank (i.e. non-character containing) lines, as well as lines that consist solely of one-or-more \h horizontal-whitespace characters:

raku -ne '.put if .subst(/^ \h+ $/).chars;'  

Sample Input:

#[blank]
#[blank]
Title 1
12345678 1234
#[whitespace] 
Title 2
12345678 1234
12345678 1234
#[blank]
Title 3
12345678 1234
12345678 1234
#[blank]
#[blank]
#[blank]

Sample Output (second code example above):

Title 1
12345678 1234
Title 2
12345678 1234
12345678 1234
Title 3
12345678 1234
12345678 1234

https://raku.org
https://rakudo.org