How to skip first, last non-blank line and blank lines from modification in a file?

Question

I have a file something like this:

H|ACCT|XEC|1|TEMP|20130215035845

849002|48|1208004|100|||1

849007|28|1208004|100|||1

T|2|3

Note that there are extra empty lines at the end of the file.

I want to replace the value of column 5 with column 4's value in all the lines except first and last non-blank line.

I cannot rely on the number of fields as the last line may have as many fields as the other ones, nor on the lines to modify always starting with a number.

I tried the code below:

awk 'BEGIN{FS="|"; OFS="|"} {$5=$4; print}' in.txt

Output is:

H|ACCT|XEC|1|1|20130215035845
||||
849002|48|1208004|100|100||1
||||
849007|28|1208004|100|100||1
||||
T|2|3||
||||
||||
||||

Expected output:

H|ACCT|XEC|1|TEMP|20130215035845|

849002|48|1208004|100|100||1

849007|28|1208004|100|100||1

T|2|3

How can I skip the first and last non-blank lines from getting changed? I also want to skip blank lines.

Answer 1

Here you go with awk and processing the file only once.

awk -F'|' 'NR==1{print;next} m && NF{print m}
    NF{l="\n"$0; $5=$4; m="\n"$0; c=0}; !NF{c++}
END{ print l; for (; i++<c;)print }' OFS='|' infile

Explanation:

Here we are skyping first line to being replace 5^th field's value with 4^th field's value, and just print it and do next.

... if it (current next line) was not empty line (at least contains one field NF), then take a backup of whole line with a \newline added l="\n"$0 first next set 5^th field's value with 4^th field's value $5=$4 and last set it to a variable m with a \newline added m="\n"$0;; There is a c variable as a counter flag and is used to determine the number of empty lines !NF{c++} if no line with at least one field seen; Otherwise c=0 will reset this counter.

Now we have modified line in m variable and m && NF{print m} will print it where in the next step awk runs and m has set and it's not on empty lines & NF (this is used to prevent duplication on printing when empty line).

At the end we are printing the untouched last line which we take backup every time before performing replacement END{ print l; ... and then number of empty lines which never seen a line with a field with looping for (; i++<c;)print }'.

That's much shorter if you don't need redundant empty lines.

awk -F'|' 'NR==1{print;next} m && NF{print m}
    NF{l=$0; $5=$4; m=$0} END{ print l}' OFS='|' infile

Answer 2

With sed, relying on the second line being blank:

sed '1{n;d;};/./!{H;$g;$p;d;};x;s/|/\n/4;s/\([^|]*\)\n[^|]*/\1|\1/'

If your sed doesn't understand \n in the replacement, use a literal newline instead (or use a character known not to be part of the file).

Explanation:

Lines (except for the first one) get collected in the hold space, when the end of the file is reached, the hold space is printed as is, otherwise with the desired replacement.

In detail:

• 1{n;d;}: For the 1st line, n prints it unchanged reading the next line, just to delete it. Why? Because the hold space is to contain something to be printed, so it contains an empty line anyhow.
• /./!{H;$g;$p;d;} is executed for empty lines only, appending itself to the Hold space. Only for the last line $ move the hold space back and print it. In any case delete to stop further executing for this line.

• x exchanges the non-empty line with the hold buffer, so it is kept there, while we can now process the saved lines, because we know it was not the last not-empty one.

• s/|/\n/4;s/\([^|]*\)\n[^|]*/\1|\1/ performs copying from column 4 to 5 by replacing the fourth | with a newline to mark it and then replace the fields before and after the match with two times the field before.

Answer 3

As I said, the easiest way is to process the file twice.
1st pass - get the line no. for the last non-empty line.
2nd pass - process all lines (except the header) before that last non-empty line that have at least five fields:

awk -F'|' -vc=0 'NR==FNR{if (NF){c=NR};next};
FNR>1 && NF>4 && FNR<c {$5=$4};1' OFS='|' infile infile

Answer 4

I made the assumption, what if the line have only four columns - the fifth column should be added, with fourth column's value. Right?

First version - awk is used

awk '
BEGIN {
    FS = "|";
    OFS = "|";
} 
FNR == NR && $0 {
    last = NR;
}
FNR != NR {
    if(NF > 3 && FNR != last && FNR != 1) {
        $5 = $4;
    }
    print;
}' input.txt input.txt

Same code with comments:

awk '
BEGIN {
    FS = "|";
    OFS = "|";
} 
# The first traversing through file
# It is needed for getting the number of the last, non-empty line
FNR == NR && $0 {
    last = NR;
}
# The second traversing through file
FNR != NR {
    # if the number of fields more than 3 (therefore, the fourth column exists)
    # and the line number of the current file is not the last and not the first. 
    if(NF > 3 && FNR != last && FNR != 1) {
        $5 = $4;
    }
    print;
}' input.txt input.txt

Second version - sed and tac are used

tac input.txt | 
sed '
1,/./!{
    $!{
        s/\(|\w*\)/\1\1/3
        s/|\w*//5
    }
}' | tac 

Explanation:

1. tac - concatenate and print files in reverse. tac is a cat in reverse.
2. 1,/./! - skip lines from the first to the first non-empty (including).
3. $! - all lines except the last. Remember that we flipped the file, and the last line is the first line in fact.
4. s/\(|\w*\)/\1\1/3 - duplicating the fourth column. I decided use \w instead of [^|] for beauty. But you can change it, if non-word characters are expected in the fields.
5. s/|\w*//5 - removes the former fifth column (now it is sixth).
6. | tac - flips the file back.