how to print the closest column when looking for a specific strings

Question

I have a file with around 12 000 lines.

pk=91001011964;fck=85;fcv=true;fpi=1;fci=cacf;fmd=1422745568,;fck=83;fcv=decoccm;fpi=1;fci=cacf;fmd=1423070648,;fck=87;fcv=false;fpi=1;fci=cacf;fmd=1422745568,;fck=86;fcv=true;fpi=1;fci=cacf;fmd=1422745568,;fck=35;fcv=54bed48ad84397433fef0a350d65a893;fpi=1;fci=cacf;fmd=1422745568,;fck=83;fcv=travide;fpi=1;fci=cacf;fmd=1422811693,;fck=88;fcv=true;fpi=1;fci=cacf;fmd=1422745568,;fck=84;fcv=10;fpi=1;fci=cacf;fmd=1422745568;vpi=ppi=1;pci=cacf;pmd=1423070648;pos=null;pod=0;pso=null;psd=0;vos=null
pk=91001045537;fck=86;fcv=true;fpi=1;fci=cacf;fmd=1421428238,;fck=85;fcv=true;fpi=1;fci=cacf;fmd=1421428238,;fck=87;fcv=false;fpi=1;fci=cacf;fmd=1421428238,;fck=88;fcv=true;fpi=1;fci=cacf;fmd=1421428238,;fck=84;fcv=10;fpi=1;fci=cacf;fmd=1421428238,;fck=35;fcv=cd9b9ed4fc167d8007dd9544114bc83f;fpi=1;fci=cacf;fmd=1421428238,;fck=83;fcv=demelog;fpi=1;fci=cacf;fmd=1421557377;vpi=ppi=1;pci=cacf;pmd=1421557377;pos=null;pod=0;pso=null;psd=0;vos=null   
pk=91001440737;fck=86;fcv=true;fpi=1;fci=cacf;fmd=1421687966,;fck=85;fcv=true;fpi=1;fci=cacf;fmd=1421687966,;fck=87;fcv=true;fpi=1;fci=cacf;fmd=1421687966,;fck=88;fcv=true;fpi=1;fci=cacf;fmd=1421687966,;fck=84;fcv=10;fpi=1;fci=cacf;fmd=1421687966,;fck=35;fcv=7360072ca14cb5d5578a3902fbe51792;fpi=1;fci=cacf;fmd=1421687191;fck=78;fcv=de724a544277d79c14d19809fe51ab71;fpi=1;fci=cacf;fmd=1421687966,;fck=83;fcv=demelog;fpi=1;fci=cacf;fmd=1421816564;vpi=ppi=1;pci=cacf;pmd=1421816564;pos=null;pod=0;pso=null;psd=0;vos=null

I need to parse the file, look for specific string values in a specific order in the line and print or pick the closest column for that line. If that string value does not exist, print an empty column.

Let's say I have fck=35. if it exists , print the searched string and the closest fmd. For line 3, it would be fck=35;fmd=1421687191. If fck=35 does not exist, add an empty column and that would make fck=35;;

Let's say I'm looking after the for fck=78. If it does exist, print the searched string and the closest fmd. For line 3, I will have fck=78;fmd=1421687966. If fck=78 does not exist, add an empty column and that would make fck=78;;.

The other conditions are the following

For fck=84 look for the next fcv For fck=85 look for the next fcv For fck=86 look for the next fcv For fck=83 look for the next fcv The next fcv for fck=83 can be naismc, decoide, decoccm, travide, travccm or mariccm

I will use line 3 as my example :

fck=35;fmd=1421687191;fck=78;fmd=1421687966;fck=84;fcv=10;fck=85;fcv=true;fck=86;fcv=true;fck=83;fcv=demelog

or (in case fmd or fcv are missing or at least some: this part is no longer line 3, only another line I've not put in my example but I guess it does not matter)

fck=35;;fck=78;;fck=84;fcv=10;fck=85;fcv=true;fck=86;;fck=83;fcv=demelog

I was thinking to do this in awk or sed. But I have a doubt on how powerful it is and for what I have seen so far, it is not helpful. Maybe Python can do the trick but I'm quite unfamiliar with Python.

Any help is welcome.

A mate of mine helped me out with this

awk -F\; -v fck=78 '{for (i=1;i<=NF;i++) if($i=="fck="fck) print "fck="fck";"$(i-1)}' name_of_the_file

As you can see, it is only taking one condition , here fck=78 when I need to have all the conditions above.

Any hints on what I can improve on that tiny script or anything that can make a difference would be great. But another script is possible. I was thinking about Python but I'm not sure on how to do that.

TIA

Answer 1

I assume you have a file with the various strings you are looking for. Something like this:

fck=35 fmd
fck=78 fcv
bnv=12 fcv

For each of those, you want to search the file and if a line matches any of the patterns, you want the first fmd value following the matched string. If so, I would do it in perl as follows:

#!/usr/bin/env perl

## Open the list of search patterns.
## The script expects it to be the 1st argument.
open(my $list,"$ARGV[0]");
## Read the file and save the patterns
## in the %pat hash.
while (<$list>) {
    ## remove trailing newlines
    chomp;
    ## separate the search pattern from the target
    my @fields=split(/\s+/);

    ## Save the search pattern and accompanying target in
    ## in the hash (%pats). 
    $pats{$fields[0]}=$fields[1];
}

## Open the list of search patterns.
## The script expects it to be the 2nd argument.
open(my $file,"$ARGV[1]");

## Read the file
while (<$file>) {
    ## split the line on ';' into the @fields array
    my @fields=split(/;/);

    ## This is the string that will be printed for
    ## the current line.
    my $outstring="";
    ## Check each of the search patterns against
    ## each of the fields.
    foreach my $pat(keys(%pats)) {
        ## Add the pattern to the outstring
        $outstring.="$pat;";
        ## save all all 1st fmd values that follow
        ## this pattern. 
        my @matches= ( /$pat.+?($pats{$pat}=[^;]+)/g );
        ## Add this pattern's matches to the output string.
        $outstring.= join(";",@matches) . ";";
    }
    ## Print the output string for this line
    print "$outstring\n";
}

If you save the script above as, for example, parser.pl in your $PATH and make it executable (chmod 755 ~/bin/parser.pl), you can run it like this:

$ parser.pl list.txt file.txt 
bnv=12;;fck=35;fmd=1422745568,;fck=78;;
bnv=12;;fck=35;fmd=1421428238,;fck=78;;
bnv=12;;fck=35;fmd=1421687191 fmd=1111111111;fck=78;fcv=de724a544277d79c14d19809fe51ab71;

Answer 2

If you're looking for the closest column and know the delimiter this should be an easy task for grep and sed.

grep -e "fck=35" 

Will return the whole line fck=35 is on. Then pipe it to two seds to get what you want.

grep -e "fck=35" | sed s/.*fck=35;//g | sed s/;.*//g

the first sed replaces everything before fck=35; and fck=35; itself with nothing (removes it), the second sed removes everything after the next delimiter.

But it sounds like you also want to be able to select a specific column (fmd) on the line, for that you need something more like:

grep -e "fck=35" | sed s/.*fmd=/fmd=/g | sed s/[;,].*//g

This removes all the elements before 'fmd=', then removes everything after the next delimiter (or comma, seems like you need to handle that).