5

I have the following situation, with two files.

File 1

not relevant = does not matter
some stuff
# var1=1
# var 2 = 2
# var3 = 3
some stuff

File 2

some other stuff
# does not matter either
# var1=a
# var 2 = b
# var3 = c
some other stuff

I would like to extract with a bash script from all lines starting with # and containing a = the corresponding variable and its value, collect all and write it to a new file

var1,var 2,var3
1,2,3
a,b,c

The variable names can contain spaces. Also before and after the = a space could appear. The values on the right hand side do not contain spaces. If a space appears it does so in both files.

2
  • 1
    The variable are the same in both files? If In file1 there are var1 , var2, var3 and var4 then the same will be in file2 too? Jul 2 at 7:50
  • 1
    How many times a variable can appears in a file? Jul 2 at 8:03

6 Answers 6

5

This is a task that needs a text processing language (such as awk or perl), not a shell script.

$ cat vars2csv.pl 
#!/usr/bin/perl

use strict;

# %vars is a Hash-of-Hashes (HoH) where the primary keys
# are the filenames, and each element is a hash containing
# each "variable" name found in in the input and its
# corresponding value.  See man pages for perldata and perldsc.
my %vars;

# Array @fields and hash %seen to keep track of new
# "variable" names in the order we see them.
my @fields;
my %seen;

# Keep a copy of the arguments so we can output the data in
# the same order we read them.
my @files = @ARGV;

while (<>) {
  chomp;
  next unless /^#.*=/;
  s/^#\s*//;

  my ($key,$val) = split /\s*=\s*/, $_, 2;

  if (!defined($seen{$key})) {
    push @fields, $key;
    $seen{$key} = 1;
  };

  # $ARGV is the name of the current file being read
  # by the `while(<>)` loop.
  $vars{$ARGV}{$key} = $val;
};

print join(",", @fields), "\n";

foreach my $f (@files) {
  next unless -r $f;  # skip output for filenames that weren't readable
  print join(",", @{$vars{$f}}{@fields}), "\n";
};

The script keeps track of the order that files were read and that field names were seen because perl hashes are inherently un-ordered (this is common for most implementations of associative arrays in most languages). I could have written it to sort the keys during the output stage (perl has a very useful built-in sort function), so at least they would be output in a predictable order, but I thought it better to use a few variables to remember the original order.

It works with any number of output fields and doesn't care what the field names or the values are. In a matching line, everything after any leading whitespace and before the first = sign is the "key", and everything after the first = sign is the value. Whitespace around the = is not included in either key or value (the line is split on \s*=\s* rather than just =). see perldoc -f split for details on the split function.

If a given key appears more than once in a file, then the value in the last occurrence will be the one that is output. If you want it to keep the first and ignore any subsequent occurrence(s), add the following line before the $vars{$ARGV}{$key} = $val; line:

next if (defined($vars{$ARGV}{$key}));

Sample run:

$ chmod +x ./vars2csv.pl

$ ./vars2csv.pl file1 file2 
var1,var 2,var3
1,2,3
a,b,c

Worth noting: this script ignores all lines that don't start with a # and contain an =. That means it processes ALL lines matching that criterion - including any comment lines that co-incidentally happen to contain an = that you didn't intend to define a variable. Depending on what, exactly, is in your input files, this may be a bug that needs to be fixed (by figuring out either a pattern to exclude these unwanted lines, or by crafting a better pattern to match only wanted lines).


BTW, I added the next unless -r $f; line in the script because I tested the script with filename args that didn't exist and with permissions that prevented them from being read. Perl printed a warning message when such errors occurred, but the script printed a line with empty fields separated by commas. This line prevents that output.

The script will also print a line of empty fields separated by commas for readable files that didn't contain any var=value comments. If you want to prevent output for those files too, add the following before the print join... line.

  next unless (keys %{ $vars{$f} }); # skip output for files with NO key=val comments

Files with some but not all fields will be printed with correct values for the fields they do have and empty values for any missing fields. e.g. a file with only # var1=1 would print 1,, as the output line. If you want to skip output for these files:

  next unless (@{$vars{$f}}{@fields}); # skip output for files missing ANY key
0
3

Use with any awk in any shell :

#!/usr/bin/awk -f
BEGIN {FS = " ?= ?" ; OFS="," ;}
NF == 2 && /^#/ {
    sub(/^# /, "", $1)

    if (FILENAME != oldFileName) {
        files[filesCnt++] = FILENAME
        oldFileName = FILENAME
    }

    hdrYetFoundIdx = -1
    for (i = 0; i < hdrCnt; i++) {
        if (hdr[i] == $1) {
            hdrYetFoundIdx = i
            break
        }
    }
    if (hdrYetFoundIdx == -1) hdr[hdrCnt++] = $1
    val[files[filesCnt-1],$1] = $2
}

END {
    for (i = 0; i < hdrCnt; i++) 
        printf "%s%s", hdr[i], ((i<hdrCnt-1)?OFS:ORS)

    for (i = 0; i < filesCnt; i++)
        for (j = 0; j < hdrCnt; j++)
            printf "%s%s", val[files[i],hdr[j]], ((j<hdrCnt-1)?OFS:ORS)
}

Note that if spaces can be between the start of the line and # then the awk condition to process code need to become : NF == 2 && /^ *#/ and the call to sub become : sub(/^ *#/, "", $1)

4
  • 1
    Fields, strings, and generated arrays start at 1 in awk, not 0, so you should try to get into the habit of starting your user-defined arrays at 1 instead of 0 too to avoid the potential for confusion and mistakes when coding array loops, etc.
    – Ed Morton
    Jul 3 at 2:06
  • 1
    @EdMorton I never ever seen that in documentation, but seem reasonable. I admit it's a bit odd to set hdrYetFoundIdx to -1 to check if an header was encounter before. Jul 3 at 16:06
  • Yeah the usual way to do what you're doing with hdrYetFoundIdx = -1 for (i = 0; i < hdrCnt; i++) { if (hdr[i] == $1) { hdrYetFoundIdx = i break } } if (hdrYetFoundIdx == -1) hdr[hdrCnt++] = $1 is just if ( !seen[$1]++ ) hdr[++hdrCnt] = $1.
    – Ed Morton
    Jul 3 at 17:11
  • And if you changed files[filesCnt++] = FILENAME to files[++filesCnt] = FILENAME then your later code becomes simpler, e.g. instead of val[files[filesCnt-1],$1] = $2 it's just val[files[filesCnt],$1] = $2 and instead of for (i = 0; i < hdrCnt; i++) printf "%s%s", hdr[i], ((i<hdrCnt-1)?OFS:ORS) it's just for (i = 1; i <= hdrCnt; i++) printf "%s%s", hdr[i], (i<hdrCnt?OFS:ORS), etc. (no need to subtract 1 in multiple places).
    – Ed Morton
    Jul 3 at 17:15
2

Assuming first file defines the set of vars to get:

#!/usr/bin/perl

use strict;
$/=undef;                                  # no input register separator
$,="|";                                    # ouput field separator
$\="\n";                                   # ouput register separa
my (%pair,@var);

while(<>){                                 # for each file
   my %pair= (m/#\s*(\S.*?)\s*=\s*(.*)/g); # get the (var->value) pairs
   if(not @var){
      @var = keys(%pair);                  # get and print schema 
      print( @var );
   }
   print( @pair{@var} );                   # print the values
}
2

This awk uses bidim array, and adds a column to mention the origin file name. Tested with gawk, mawk and in POSIX mode. It tolerates variables that are not in all input files, any number of input files, any values not containing ",".

#!/usr/bin/env -S awk -f
BEGIN{ OFS="," } # define output field separator
/^#.+=/ && $0 !~ OFS { # safe input lines
  sub("^# *", "") # trim left
  sub(" *$", "") # trim right
  sub(" *= *", OFS) # trim and replace sep
  num = split($0, val, OFS) # split
  if (num == 2 && val[1] && val[2]) { # exactly 2 non-empty values
    files[FILENAME] = 1
    names[val[1]] = 1
    values[val[1],FILENAME] = val[2]
  }
}  
END { # output results
  PROCINFO["sorted_in"]="@ind_str_asc" # preferred order
  $0="" # empty record
  for (name in names) { # stable order
    $++NF=name
  }
  $++NF="from"
  print
  for (file in files) {
    $0="" # empty record
    for (name in names) { # stable order
      $++NF=values[name,file]
    }
    $++NF=file
    print
  }
}

Output:

var 2,var1,var3,from
2,1,3,file1
b,a,c,file2
5
  • there are many features available only to GNU awk Jul 2 at 10:44
  • On my system nawk is gawk. Jul 2 at 11:09
  • still gawk. PROCINFO is a gawk features Jul 2 at 11:31
  • @DanieleGrassini yes, anyway mawk doesn't suffer this line and its order is the same as gawk's ind_str_asc, so that they both give the same output. However the column order is not significant, order stability is only required locally. Jul 2 at 11:38
  • Regarding its order is the same as gawk's ind_str_asc - that's just coincidence given some specific input you used. Different input would produce output in a different order. I'm not saying that's a problem.
    – Ed Morton
    Jul 3 at 11:31
1

Assuming all records of all files have three variable assignments for the same three variables in the same order, with pcregrep:

assign='#.*?=\h*(.*?)\h*'
pcregrep -hMo1 -o2 -o3 --om-separator=, "^$assign\n$assign\n$assign\$" file1 file2 

Would give you the values. For the header, you can extract it from the first record in the first file:

assign='#\h*(.*?)\h*=.*'
pcregrep -Mo1 -o2 -o3 --om-separator=, "^$assign\n$assign\n$assign\$" file1 | head -n1
1

Using any POSIX awk:

$ cat tst.awk
BEGIN { OFS="," }
sub(/^#[[:space:]]*/,"") && /=/ {
    tag = val = $0
    sub(/[[:space:]]*=.*/,"",tag)
    sub(/[^=]+=[[:space:]]*/,"",val)
    vals[FILENAME,tag] = val
    if ( !seen[tag]++ ) {
        tags[++numTags] = tag
    }
}
END {
    for ( tagNr=1; tagNr<=numTags; tagNr++ ) {
        tag = tags[tagNr]
        printf "%s%s", tag, (tagNr<numTags ? OFS : ORS)
    }

    for ( fileNr=1; fileNr<ARGC; fileNr++ ) {
        fileName = ARGV[fileNr]
        for ( tagNr=1; tagNr<=numTags; tagNr++ ) {
            tag = tags[tagNr]
            printf "%s%s", vals[fileName,tag], (tagNr<numTags ? OFS : ORS)
        }
    }
}

$ awk -f tst.awk file1 file2
var1,var 2,var3
1,2,3
a,b,c

If you don't have a POSIX awk just change [:space:] to [ \t] and it'll work in any awk if all the white space in input lines is just blanks or tabs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.