3

I have a file something like this:

H|ACCT|XEC|1|TEMP|20130215035845

849002|48|1208004|100|||1

849007|28|1208004|100|||1

T|2|3



Note that there are extra empty lines at the end of the file.

I want to replace the value of column 5 with column 4's value in all the lines except first and last non-blank line.

I cannot rely on the number of fields as the last line may have as many fields as the other ones, nor on the lines to modify always starting with a number.

I tried the code below:

awk 'BEGIN{FS="|"; OFS="|"} {$5=$4; print}' in.txt

Output is:

H|ACCT|XEC|1|1|20130215035845
||||
849002|48|1208004|100|100||1
||||
849007|28|1208004|100|100||1
||||
T|2|3||
||||
||||
||||

Expected output:

H|ACCT|XEC|1|TEMP|20130215035845|

849002|48|1208004|100|100||1

849007|28|1208004|100|100||1

T|2|3



How can I skip the first and last non-blank lines from getting changed? I also want to skip blank lines.

  • 2
    How can we know the last line? Will there always be the same number of lines? Your output suggests that your input file has several blank lines at the end. – terdon Oct 13 '17 at 12:44
  • @terdon : Max will be 3 blank line in the last, that is the main challenge. In question from last line I mean that line which is not blank in the last. – serenesat Oct 13 '17 at 12:49
  • ok... from your sample only lines starting with a number require to be changed... can that be used in solution? because restricting last non-blank line is not trivial – Sundeep Oct 13 '17 at 13:01
  • @Sundeep: No, that can't be used. First line and last non-blank line can also be start with number. – serenesat Oct 13 '17 at 13:06
  • @don_crissti yup that would make logic easier :) – Sundeep Oct 13 '17 at 13:07
5

Here you go with awk and processing the file only once.

awk -F'|' 'NR==1{print;next} m && NF{print m}
    NF{l="\n"$0; $5=$4; m="\n"$0; c=0}; !NF{c++}
END{ print l; for (; i++<c;)print }' OFS='|' infile

Explanation:

Here we are skyping first line to being replace 5th field's value with 4th field's value, and just print it and do next.

... if it (current next line) was not empty line (at least contains one field NF), then take a backup of whole line with a \newline added l="\n"$0 first next set 5th field's value with 4th field's value $5=$4 and last set it to a variable m with a \newline added m="\n"$0;; There is a c variable as a counter flag and is used to determine the number of empty lines !NF{c++} if no line with at least one field seen; Otherwise c=0 will reset this counter.

Now we have modified line in m variable and m && NF{print m} will print it where in the next step awk runs and m has set and it's not on empty lines & NF (this is used to prevent duplication on printing when empty line).

At the end we are printing the untouched last line which we take backup every time before performing replacement END{ print l; ... and then number of empty lines which never seen a line with a field with looping for (; i++<c;)print }'.

That's much shorter if you don't need redundant empty lines.

awk -F'|' 'NR==1{print;next} m && NF{print m}
    NF{l=$0; $5=$4; m=$0} END{ print l}' OFS='|' infile
  • First code is not changing the value. – serenesat Oct 18 '17 at 10:11
  • Hi @serenesat It is changing! Both codes does the changes. – αғsнιη Oct 18 '17 at 10:19
  • Nope. second one is changing but not the first one. First one doesn't make any change and also doesn't show any error. – serenesat Oct 18 '17 at 10:21
3

With sed, relying on the second line being blank:

sed '1{n;d;};/./!{H;$g;$p;d;};x;s/|/\n/4;s/\([^|]*\)\n[^|]*/\1|\1/'

If your sed doesn't understand \n in the replacement, use a literal newline instead (or use a character known not to be part of the file).

Explanation:

Lines (except for the first one) get collected in the hold space, when the end of the file is reached, the hold space is printed as is, otherwise with the desired replacement.

In detail:

  • 1{n;d;}: For the 1st line, n prints it unchanged reading the next line, just to delete it. Why? Because the hold space is to contain something to be printed, so it contains an empty line anyhow.
  • /./!{H;$g;$p;d;} is executed for empty lines only, appending itself to the Hold space. Only for the last line $ move the hold space back and print it. In any case delete to stop further executing for this line.
  • x exchanges the non-empty line with the hold buffer, so it is kept there, while we can now process the saved lines, because we know it was not the last not-empty one.

  • s/|/\n/4;s/\([^|]*\)\n[^|]*/\1|\1/ performs copying from column 4 to 5 by replacing the fourth | with a newline to mark it and then replace the fields before and after the match with two times the field before.

  • 1
    Nice! And yes, please explain that. This sort of complex sed commands look like Asterix swearing. – terdon Oct 13 '17 at 14:35
  • I like the term "Asterix swearing", although it better fits to some crazy languages on code golfing. Anyhow, added the explanation. – Philippos Oct 13 '17 at 14:56
3

As I said, the easiest way is to process the file twice.
1st pass - get the line no. for the last non-empty line.
2nd pass - process all lines (except the header) before that last non-empty line that have at least five fields:

awk -F'|' -vc=0 'NR==FNR{if (NF){c=NR};next};
FNR>1 && NF>4 && FNR<c {$5=$4};1' OFS='|' infile infile
0

I made the assumption, what if the line have only four columns - the fifth column should be added, with fourth column's value. Right?

First version - awk is used

awk '
BEGIN {
    FS = "|";
    OFS = "|";
} 
FNR == NR && $0 {
    last = NR;
}
FNR != NR {
    if(NF > 3 && FNR != last && FNR != 1) {
        $5 = $4;
    }
    print;
}' input.txt input.txt

Same code with comments:

awk '
BEGIN {
    FS = "|";
    OFS = "|";
} 
# The first traversing through file
# It is needed for getting the number of the last, non-empty line
FNR == NR && $0 {
    last = NR;
}
# The second traversing through file
FNR != NR {
    # if the number of fields more than 3 (therefore, the fourth column exists)
    # and the line number of the current file is not the last and not the first. 
    if(NF > 3 && FNR != last && FNR != 1) {
        $5 = $4;
    }
    print;
}' input.txt input.txt

Second version - sed and tac are used

tac input.txt | 
sed '
1,/./!{
    $!{
        s/\(|\w*\)/\1\1/3
        s/|\w*//5
    }
}' | tac 

Explanation:

  1. tac - concatenate and print files in reverse. tac is a cat in reverse.
  2. 1,/./! - skip lines from the first to the first non-empty (including).
  3. $! - all lines except the last. Remember that we flipped the file, and the last line is the first line in fact.
  4. s/\(|\w*\)/\1\1/3 - duplicating the fourth column. I decided use \w instead of [^|] for beauty. But you can change it, if non-word characters are expected in the fields.
  5. s/|\w*//5 - removes the former fifth column (now it is sixth).
  6. | tac - flips the file back.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.