0

I want to create a bash script which iterates through a file, one line at a time, and produces a coherent output:

example.txt

ALBERT some a BRYAN some b CLAUDIA some c DAVID some d ERIK some e
ALBERT some a BRYAN some b ERIK some e
ALBERT some a BRYAN some b DAVID some d

some notes:

  • the word count between the tags is varying
  • the keywords always appear in the same order
  • A complete list of keywords is available and could be specified before hands

Needed output:

some a; some b; some c; some d; some e
some a; some b;;; some e
some a; some b;; some d;

I can easily replace the keywords one by one with a semicolon using sed:

sed -i 's/ALBERT/;/g' "example.txt"

How can I use awk to iterate through every line and add the required semicolons if some of the keywords are missing? I suspect one has to introduce some kind of counter?

7
  • 1. do you have a complete list of keywords? or can they only be discovered by processing the input file? 2. is "some x" always two words, or can it be text of any arbitrary length?
    – cas
    Apr 19, 2021 at 16:32
  • @cas yes, I have a complete list of keywords. They could also be discovered by scanning for words in CAPS but thats not necesserily needed for my use case. "some x" can be text of any arbitrary length, which makes this tasks a little bit more complex. I assume one has to scan the line for a keyword (they always appear in the same order) and if its not found, increase a counter. Than insert the needed amount of semicolons for the next found keyword.
    – losttoy
    Apr 19, 2021 at 16:46
  • how do you determine the order of the output fields on each line of output? alphabetic sort? e.g. if scanning the file, you find keys A, B, C on line 1, then new keys E and F on line 2, and key D on line 3 - are the fields output in sorted order ABCDEF, or in discovered order ABCEFD? I presume sorted. Of course, if the keywords are known in advance, a fixed order can be used.
    – cas
    Apr 19, 2021 at 16:50
  • also, do you care if the output is aligned in columns visually, or only care that there is a semi-colon separating fields? Also, why does your sample output NOT have some e for the third output line? is that intended or a mistake?
    – cas
    Apr 19, 2021 at 16:53
  • @cas only the semicolon is important to have the parsed data imported into another programm. Since the keywords and their order are known in advance, a fixed order can be specified. In a single line, the keywords are also unique (not appearing multiple times in a line)
    – losttoy
    Apr 19, 2021 at 16:56

2 Answers 2

1

Assuming that some tags (names like "ALBERT") could be missing from the first line just like they can be missing from other lines, you need a 2-pass approach to first identify all of the tags and then print the values for all of them for every line whether they are present on that line or not.

$ cat tst.awk
BEGIN { OFS=";" }
NR==FNR {
    for (i=1; i<NF; i+=3 ) {
        if ( !seen[$i]++ ) {
            tags[++numTags] = $i
        }
    }
    next
}
{
    delete tag2val
    for (i=1; i<NF; i+=3) {
        tag = $i
        val = $(i+1) FS $(i+2)
        tag2val[tag] = val
    }

    for (tagNr=1; tagNr<=numTags; tagNr++) {
        tag = tags[tagNr]
        val = tag2val[tag]
        printf "%s%s", val, (tagNr<numTags ? OFS : ORS)
    }
}

$ awk -f tst.awk example.txt example.txt | column -t -s';' -o'; '
some a; some b; some c; some d; some e
some a; some b;       ;       ; some e
some a; some b;       ; some d;

The above will output for each line the values for all of the tags in the order they appeared across all of the input.

If you want to see the tags as column headers:

$ cat tst.awk
BEGIN { OFS=";" }
NR==FNR {
    for (i=1; i<NF; i+=3 ) {
        if ( !seen[$i]++ ) {
            tags[++numTags] = $i
        }
    }
    next
}
FNR==1 {
    for (tagNr=1; tagNr<=numTags; tagNr++) {
        tag = tags[tagNr]
        printf "%s%s", tag, (tagNr<numTags ? OFS : ORS)
    }
}
{
    delete tag2val
    for (i=1; i<NF; i+=3) {
        tag = $i
        val = $(i+1) FS $(i+2)
        tag2val[tag] = val
    }

    for (tagNr=1; tagNr<=numTags; tagNr++) {
        tag = tags[tagNr]
        val = tag2val[tag]
        printf "%s%s", val, (tagNr<numTags ? OFS : ORS)
    }
}

$ awk -f tst.awk example.txt example.txt | column -t -s';' -o'; '
ALBERT; BRYAN ; CLAUDIA; DAVID ; ERIK
some a; some b; some c ; some d; some e
some a; some b;        ;       ; some e
some a; some b;        ; some d;
2
  • The problem with your approach is that you asume a fixed number of words between the tags. I have varying number of words between the tags but the tags are always in CAPS and it would also be ok to specify them before hand, so I suppose another approach is needed. Btw. I also have a complete list of the tags and they appear in the same order (except the missing ones) throughout the lines.
    – losttoy
    Apr 19, 2021 at 16:48
  • 2
    We have to assume something and that looked more likely from your example than that your tags would be all upper case and the other text wouldn't be. If you have varying numbers of words in your data than show varying numbers of words in your example. If you have a list you want us to use then show that in the example in your question too. All we have to go on to help you is what you tell us about in your question and show in your example.
    – Ed Morton
    Apr 20, 2021 at 1:17
0

Using perl:

#!/usr/bin/perl

# @keys is an array containing the keywords. It also determines
# the field output order.  This can be read from a file if needed,
# but here it's hard-coded.
my @keys = qw(ALBERT BRYAN CLAUDIA DAVID ERIK);

# create and pre-compile a regex matching all the keywords
my $keys = join("|",@keys);
my $keys_re = qr/$keys/;

# make an empty hash containing elements for all the keys so that
# we can start processing each input record afresh, with a fully
# populated list of keys.
my %empty = map +( $_ => '' ), @keys;


# main loop, process stdin and/or filename args
while(<>) {
  # clean up the input a little.
  chomp;            # trim newlines at EOL
  s/^\s*|\s*$//g;   # trim leading and trailing whitespace

  # ignore empty lines.
  next if (m/^$/);

  # NUL can't be in text input, so insert it as a marker around
  # the keywords. i.e. insert NULs before and after each keyword
  s/$keys_re/\000$&\000/g;

  # split the input record on NUL, trimming spaces and discarding
  # the first element (a bogus artificial field which only exists
  # as a side-effect of inserting a NUL before the first keyword.)
  my (undef,@record) = split /\s*\000\s*/;

  # pre-populate the fields hash for each record.
  my %fields = %empty;

  # now insert the real values for each keyword if they exist.
  foreach my $i (0..$#record) {
    $fields{$record[$i]} = $record[$i+1];
    $i++;
  };

  print join(";", map +( $fields{$_} ), @keys),"\n";
}

If you want a space after each semi-colon, change the print join(";",...) line above and add one.

To read the keywords from a file, replace the my @keys = qw(...) line above with:

# slurp in the keywords file and split it on any whitespace.
my @keys = split /\s+/, do {
  local $/;   # read entire file at once - slurp
  my $fname = 'keywords.txt';
  open(my $fh, '<', $fname) or die "Error opening $fname: $!";
  <$fh>
};

keywords.txt can contain the keys separated by any combination of vertical or horizontal whitespace - spaces, tabs, newlines, and CR/LFs, etc. e.g.

$ cat keywords.txt 
ALBERT
BRYAN   CLAUDIA
DAVID ERIK

Save it as, e.g. iterate.pl, and make it executable with chmod +x iterate.pl.

$ ./iterate.pl input.txt 
some a;some b;some c;some d;some e
some a;some b;;;some e
some a;some b;;some d;

If you want prettier output for viewing in less or whatever, you can use column, e.g.

$ ./iterate.pl input.txt | column -s';' -o'; ' -t
some a; some b; some c; some d; some e
some a; some b;       ;       ; some e
some a; some b;       ; some d; 
2
  • It looks like there's (yet another) bug in SE's perl syntax formatting. Every instance of \x0 above should be \x00. That's what's in my code, but it's displaying incorrectly. The code will still work as displayed (at least, it does for the sample input), but it's wrong - \x requires two hex digits, having only one can break things.
    – cas
    Apr 19, 2021 at 18:16
  • using octal \000 instead of hex \x00 works - doesn't confuse SE's syntax highlighting.
    – cas
    Apr 20, 2021 at 2:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .