add one blank line in awk on a condition

Question

Morning everybody,

I would like to divide groups of matching lines of a file with a blank line. Being new to awk I tinkered around a bit and came up with this:

awk '!($0 in a) {print "\n"; a[$0]}; {print}'

which in my head reads as

If the current line is not in array 'a' then print a newline and add the line to 'a'. Print the current line.

If I run this against a testfile the output looks like

abc
abc


def
def
def


ghi

i.e. there are two blank lines printed instead of one. Where do the extra lines come from?

This is the test file I used:

abc
abc
def
def
def
ghi

Answer 1

You don't need for an associated array:

awk 'prev!=""{ print prev!=$0? prev ORS : $0 } { prev=$0 }
END{ if(prev!="") print prev }' infile

Output:

abc
abc

def
def
def

ghi

About why your awk is printing newline twice it's because you are using print statement, which by default it's printing what ever you are printing + ORS (Output Record Separator, which default is newline), you need to use printf "\n" instead or just print ""; and with your own solution, you could do as following (applying some fixes):

awk '!($0 in a) { if(c++) print "" } { a[$0]; print}' infile

or more compact:

awk '!($0 in a) && c++{ print ""} ++a[$0]' infile

Answer 2

$ awk '{print ($0!=p ? s : "") $0; p=$0; s=ORS}' file
abc
abc

def
def
def

ghi

The mistake in your code that's leading to 2 blank lines being printed instead of 1 was using print "\n" instead of print "", the latter being all you need to print the value of ORS:

$ awk 'BEGIN{print "---"; print "\n"; print "---"}'
---


---
$ awk 'BEGIN{print "---"; print ""; print "---"}'
---

---

You could alternatively have used printf "\n" but that's printing the hard-coded value you hope/assume that ORS has instead of simply printing ORS with print "".

The main functional differences between my solution and yours are that your script would print a blank line at the start of the output while mine wouldn't (thanks to setting s=ORS after printing the first line) and you're saving the whole of the input file in a[] while I only save 1 input line, the previous one read, in p and so:

1. Your script would use a lot of memory and so possibly fail for huge input files while mine would work for any size input file.
2. Your script would only print a blank line if any input line had never appeared anywhere previously in the input while mine would print a blank line every time the input changed so they'd behave differently from each other if the input lines weren't always grouped together, e.g.:

    $ printf 'foo\nbar\nfoo\n'
    foo
    bar
    foo

    $ printf 'foo\nbar\nfoo\n' | awk '!($0 in a) {print ""; a[$0]}; {print}'
    
    foo
    
    bar
    foo

    $ printf 'foo\nbar\nfoo\n' | awk '{print ($0!=p ? s : "") $0; p=$0; s=ORS}'
    foo
    
    bar
    
    foo

To do what you are trying to do in your code instead of an array named a[] we idiomatically name the array seen[] and update it at the same time as testing it rather than separately so your code would idiomatically have been written as awk '!seen[$0]++{print ""} 1' instead of awk '!($0 in a) {print ""; a[$0]}; {print}':

$ printf 'foo\nbar\nfoo\n' | awk '!seen[$0]++{print ""} 1'

foo

bar
foo

and if you wanted that functionality without a leading blank line in the output, take your pick:

$ printf 'foo\nbar\nfoo\n' | awk '{print (seen[$0]++ ? "" : s) $0; s=ORS}'
foo

bar
foo

$ printf 'foo\nbar\nfoo\n' | awk '!seen[$0]++ && NR>1{print ""} 1'
foo

bar
foo

$ printf 'foo\nbar\nfoo\n' | awk '!seen[$0]++{if (NR>1) print ""} 1'
foo

bar
foo

$ printf 'foo\nbar\nfoo\n' | awk '!seen[$0]++{printf s; s=ORS} 1'
foo

bar
foo

That last one would fail if ORS contained printf formatting chars, e.g.:

$ printf 'foo\nbar\nfoo\n' | awk -v ORS='\n%s\n' '!seen[$0]++{printf s; s=ORS} 1'
foo
%s
awk: cmd. line:1: (FILENAME=- FNR=2) fatal: not enough arguments to satisfy format string
        `
%s
'
          ^ ran out for this one

so you could write it more robustly as the following if that's a concern:

$ printf 'foo\nbar\nfoo\n' | awk -v ORS='\n%s\n' '!seen[$0]++{printf "%s", s; s=ORS} 1'
foo
%s

%s
bar
%s
foo
%s