3

I'm new to Unix scripting, so please bear with me.

I am given a file which has information on processes on each line. I need to extract certain information on these processes from each line.

Example of the file -

process1 port=1234 appID=dummyAppId1 authenticate=true <some more params>
process3 port=1244 authenticate=false appID=dummyAppId2 <some more params>
process2 appID=dummyAppId3 port=1235 authenticate=true <some more params>

The desired output is -

1
port=1234 authenticate=true appID=dummyAppId1 
2
port=1244 authenticate=false appID=dummyAppId2
3
port=1235 authenticate=true appID=dummyAppId3

The numbers 1, 2, and 3 on each line just denote the line number of the output file.

I have already tried using the sed s/ command but it is order-specific, while the parameters in the input file don't follow an order - as a result, some lines in the input file are skipped.

Here is my command -

sed -nr 'appId/s/(\w+).*port=([^ ]+) .*authenticate=[^ ]+) .*appId=[^ ]+) .*/\2\t\3\t\4/p' | sed =

Could anyone guide me on how to extract those parameters regardless of order?

Thanks!

Edit 1: I managed to use grep's look-behind zero-width assertion feature this way -

grep -Po '(?<=pattern1=)[^ ,]+|(?<=pattern2=)[^ ,]+|(?<=pattern3=)[^ ,]+|(?<=pattern4=)[^ ,]+' filename

but this seems to give the output for each line in new lines i.e.

1234
true
dummyAppId1

Trying to figure out how to get it on one line using grep (i.e. not via merging X lines into 1)

Edit 2: mixed-up the order of parameters in the input

Edit 3: I'm sorry, I should have mentioned this earlier - perl seems to be restricted on the machines I work on. While the answers provided by Stephane and Sundeep work perfectly when I test it out locally, it wouldn't work on the machines I need it to finally run on. It looks like awk, grep, and sed are the mainly supported options :(

0
8

With awk (tested with GNU awk, not sure if it works with other implementations)

$ cat kv.awk
/appID/ {
    for (i = 1; i <= NF; i++) {
        $i ~ /^port=/ && (a = $i)
        $i ~ /^authenticate=/ && (b = $i)
        $i ~ /^appID=/ && (c = $i)
    }
    print NR "\n" a, b, c
}

$ awk -v OFS='\t' -f kv.awk ip.txt
1
port=1234   authenticate=true   appID=dummyAppId1
2
port=1244   authenticate=false  appID=dummyAppId2
3
port=1235   authenticate=true   appID=dummyAppId3


With perl

$ # note that the order is changed for second line here
$ cat ip.txt
process1 port=1234 authenticate=true appID=dummyAppId1 <some more params>
process3 port=1244 appID=dummyAppId2 authenticate=false <some more params>
process2 port=1235 authenticate=true appID=dummyAppId3 <some more params>

$ perl -lpe 's/(?=.*(port=[^ ]+))(?=.*(authenticate=[^ ]+))(?=.*(appID=[^ ]+)).*/$1\t$2\t$3/; print $.' ip.txt 
1
port=1234   authenticate=true   appID=dummyAppId1
2
port=1244   authenticate=false  appID=dummyAppId2
3
port=1235   authenticate=true   appID=dummyAppId3
  • (?=.*(port=[^ ]+)) first capture group for port
  • (?=.*(authenticate=[^ ]+)) second capture group for authenticate and so on
  • print $. for line number
  • To avoid partial matches, use \bport, \bappID etc if word boundary is enough. Otherwise, use (?<!\S)(port=[^ ]+) to restrict based on whitespace.

If you need to print only lines containing appID or any other such condition, change -lpe to -lne and change print $. to print "$.\n$_" if /appID/

0
7

With perl, you could use an approach like:

perl -lne 'my %h;
           $h{$1} = $& while /(\S+?)=(\S+)/g;
           print "@h{qw(port authenticate appID)}"'

Where you build a hash table whose keys are the attribute names and values are the name=values and print the ones you want afterwards.

Replace $& with $2 if you only want the values on output.

Same with awk:

awk '
  {
    split("", h)
    for (i = 1; i <= NF; i++)
      if (n = index($i, "=")) h[substr($i, 1, n - 1)] = $i
    print h["port"], h["authenticate"], h["appID"]
  }'

With pcregrep, you can do:

pcregrep -o1 -o2 -o3 --om-separator=' ' '(?x)
  ^(?=.*?\s(port=\S+))
   (?=.*?\s(authenticate=\S+))
   (?=.*?\s(appID=\S+))'

(that one requires all three attributes to be present).

With sed:

sed 'G
     s/[[:space:]]\(port=[^[:space:]]*\).*\n.*/&\1/
     s/[[:space:]]\(authenticate=[^[:space:]]*\).*\n.*/& \1/
     s/[[:space:]]\(appID=[^[:space:]]*\).*\n.*/& \1/
     s/.*\n//'

Those last two assume the attributes are not the first word of the line (which seems like a reasonable assumption given your sample).

1

Complying to your EDIT 3, I think you could still do it with sed if you made a s/// expression for each parameter like this:

sed -nE 's/^(.*)(appID=[^[:blank:]]+\s)(.*)$/\2\t\1\3/
         s/^(.*)(authenticate=[^[:blank:]]+\s)(.*)$/\2\t\1\3/
         s/^(.*)(port=[^[:blank:]]+\s)(.*)$/\2\t\1\3/
         T;=
         s/^(([^[:blank:]]+\s+){,3}).*/\1/
         p'

Note the reverse ordering of the s expressions with respect to the desired order of output. The numbering is also embedded in the script, printing output line numbers as you mentioned, and it prints a line only if any one of the wanted parameters are actually present in a line. Note also that I'm leveraging GNU sed syntax as you have been using \d atoms which AFAIK are not known to BSD sed. A POSIX compliant equivalent might be possible but would likely be more expanded.

However, that is already horribly long and would become increasingly complex at the increase of the parameters to output, so an awk script like the below might be more versatile:

awk '
    BEGIN {ac=ARGC; ARGC=0; OFS="\t"}
    {
        str=$0; NF=0
        for (i=1; i<ac; i++)
            if (match(str, ARGV[i]"=[^[:blank:]]*"))
                $(NF+1)=substr(str, RSTART, RLENGTH)
    }
    NF {print ++nr; print}
    ' -- port authenticate appID

You would specify the exact parameters you want to output, and their order of appearance, as arguments to the awk script itself after the --. This script too prints a line only when at least one of the wanted parameters is actually present in a line.

0
1

Any time there are name=value pairs in the input I find it best to first create an array that holds that mapping (f[]) below and then you can access the values by their names in whatever order you like, e.g.:

$ awk -F'[ =]' '{
    for (i=2;i<NF;i+=2) f[$i]=$i"="$(i+1)
    print f["port"], f["authenticate"], f["appID"]
}' file
port=1234 authenticate=true appID=dummyAppId1
port=1244 authenticate=false appID=dummyAppId2
port=1235 authenticate=true appID=dummyAppId3
0

If it can help other users with a similar problem, a (verbose) proposal using Ruby:

# passing the log file as parameter
lines = File.open(ARGV[0]).read.split("\n")

lines.each_with_index do |line, i|
  words  = line.split(' ')
  output = []

  puts i + 1
  output << words.select { |w| w =~ /port=\d+/ }
  output << words.select { |w| w =~ /authenticate=\w+/ }
  output << words.select { |w| w =~ /appID=\w+/ }

  puts output.join(' ')
end
-1

Others have provided awk and perl solutions.

To be different, I propose a solution which uses a bash associative array:

#!/bin/bash

declare -A arr
lineno=0

while IFS=' =' read process key1 value1 key2 value2 key3 value3 restofline
do
   arr=( [$key1]="$value1" [$key2]="$value2" [$key3]="$value3" )
   printf "%d\nport=%s authenticate=%s appID=%s\n" \
          $((++lineno)) ${arr["port"]} ${arr["authenticate"]} ${arr["appID"]}
done < infile

which outputs:

$ ./demo
1
port=1234 authenticate=true appID=dummyAppId1
2
port=1244 authenticate=false appID=dummyAppId2
3
port=1235 authenticate=true appID=dummyAppId3
$

This assumes the records in the file are laid out as per the sample supplied by the OP in their question.

Some may argue that the above script is not a general solution and is therefore somehow an inferior solution. I would argue that not all solutions to a problem need to be general solution. Indeed, often times, a simple precise solution is better.

Here is a more general solution:

#!/bin/bash

declare arr1
declare -A arr2

while read -ra arr1; do
    for str in "${arr1[@]}"; do
        if [[ "$str" =~ ..*=..* ]]; then
            key=${str%=*}; val=${str#*=}; arr2[$key]="$val"
        fi
    done
    printf "port=%s authenticate=%s appID=%s\n" \
           ${arr2["port"]} ${arr2["authenticate"]} ${arr2["appID"]}
done < infile

which outputs:

$ ./demo
port=1234 authenticate=true appID=dummyAppId1
port=1244 authenticate=false appID=dummyAppId2
port=1235 authenticate=true appID=dummyAppId3
$
2
  • annoyingly non-general, and note even the question had "<some more params>" at the end of the line, so it might good to prepare for seeing more than three values on a line. You might be able to do that by reading the keys and values to an array with IFS=' =' read -a arr, and then loop over that. Or safer, just read -a arr and then split the received words again to account for the chance there are words without an equal sign in the middle – ilkkachu Aug 20 '20 at 19:35
  • @ilkkachu, the OP provided a very specific scenario. My answer addresses that very specific scenario, and is not intended to address all possible scenarios. Yes, I could have used read -a arr but I choose not to. Feel free to offer a superior bash solution. – fpmurphy Aug 20 '20 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.