How does `awk 'NF {p=1} p'` remove blank lines from the beginning and end of a file?

Question

Searching for a way to remove blank lines from the beginning and the end (using tac) of a file, I've stumbled across this one:

awk 'NF {p=1} p'

How / why does this work?

I understand NF is only true if there are any fields (if the line is not a blank line).

Answer 1

This will remove blank lines from the beginning, but not from the end of a file. [Notice: this answer was witten before the edit to the question that mentioned tac]

It works as follows:

• NF is the number of fields found on the current line. If it is zero, that means the line is either empty or blank, i.e. contains at most whitespace (assuming the field separator is left at its default value, where any number of consecutive whitspace is considered as separator).
• The current line is printed if any condition outside of (and not associated with) rule blocks ({ ... }) evaluates to true. The flag p is initially uninitialized and will evaluate to false, so a priori nothing will be printed.
• Once a non-blank line is found (NF is non-zero and evaluates to true) the rule block {p=1} is entered and the flag p set to 1. After that, the p outside the rule block evaluates to true, and any subsequent lines (including the current, first non-blank one) is printed.

Notice that since the flag p is never reset, any blank lines coming after the first non-blank line will be printed without filtering. If you want to remove blank lines from the end, too, a two-pass approach will be necessary:

awk 'FNR==NR{if (NF) {if (!first) first=FNR; last=FNR} next}
     FNR>=first && FNR<=last' input.txt input.txt

This will process the file twice (hence it is specified twice as operand)

• In the first pass, where FNR, the per-file line counter is equal to NR, the global line counter, we identify the first and last non-blank line.
• In the second pass (FNR is now smaller than NR), we only print lines between (and including) the so identified first and last non-blank lines.

Notice

As stated in the answer by Stéphane Chazelas, the two-pass approach only works with regular files. If your input is of a different nature, see the method proposed there for a solution.

Answer 2

Using this technique to remove blank lines from both the head and tail of the file:

awk 'NF {p=1} p' file | # remove blank lines at the file head
  tac |                 # reverse the lines
  awk 'NF {p=1} p' |    # remove blanks from the "new head"
  tac |                 # re-reverse the file
  sponge file           # from the `moreutils` package, to overwrite the file

Answer 3

With GNU awk for multi-char RS and \s shorthand for [[:space:]]:

awk -v RS='^\\s*\n|\n\\s*$' '$0!=""'

For example:

$ cat file


foo
bar

$ awk -v RS='^\\s*\n|\n\\s*$' '$0!=""' file
foo
bar

The above sets the record separator to blank lines (including lines that only contain space chars) at the start of the input or blank lines at the end of the input and just prints what's in between. The $0!="" is necessary because if there are blank lines at the start of the file then that'll match ^\s*\n and so awk will understand there to be an empty record before that appearance of a RS.

wrt awk 'NF {p=1} p' and How / why does this work? from your question - to be clear, it doesn't work. It'll only remove blank lines from the beginning of a file, not from the end of a file. It sets a flag (poorly named p for print instead of f for found or similar) the first time a non-empty line is found and only prints when p is true thereby not printing anything before that first non-empty line.

Answer 4

What your code does and why it only deletes blank lines at the start of the input has already been explained in @AdminBee's answer for instance, but here for completeness, I'll suggest an alternative method for removing both leading and trailing blank lines without having to make two passes on the file (which would only work for regular files and not arbitrary input).

awk '
       NF {print saved $0; saved = ""; started = 1; next}
  started {saved = saved $0 ORS}' < file

Where we delay the printing of blanks lines up to the next non-blank line we see afterwards (provided we've already seen at least one non-blank line before).

Answer 5

In case you don't mind clobbering any spaces or tabs on the blank lines that you want to keep, this will remove blank lines from beginning & end:

awk 'NF{for(;c;--c)print "";print;x=1;next} x{++c}'

It counts how many blank lines occur between non blank lines, and prints that many empty lines before each non blank line.

Answer 6

We can delete both the leading and trailing empty lines by means of the following POSIX-ly sed code. The leading empty lines are deleted by means of the range operator, which is triggered only when it sees an nonempty line. For the trailing lines, we start accumulating empty lines till we either see an eof (wherein we promptly delete this bunch of trailing empty lines) or hit an nonempty line (wherein we print this bunch of atleast one empty + exactly one nonempty line).

sed -e '
  /./,/^$/!d
  /./b
  :a
    $d;N
  /\n$/ba
' file

Answer 7

sample file

foo = 1700; goo = 1800
pravee

ajay
koy
ram

Empty lines at 1,4,8

Since we need to delete empty lines at first line and last line

a. first we find last occurence of empty line using below command and assign to variable

lastemptyline=$(awk '/^$/{x=$0}END{print NR}' file)

b. then we will use sed command to delete first and last line

sed -i -e "1{/^$/d}" -e "$lastemptyline{/^$/d}" file

output

foo = 1700; goo = 1800
pravee
    
ajay
koy
ram