This is a task that needs a text processing language (such as awk or perl), not a shell script.
$ cat vars2csv.pl
#!/usr/bin/perl
use strict;
# %vars is a Hash-of-Hashes (HoH) where the primary keys
# are the filenames, and each element is a hash containing
# each "variable" name found in in the input and its
# corresponding value. See man pages for perldata and perldsc.
my %vars;
# Array @fields and hash %seen to keep track of new
# "variable" names in the order we see them.
my @fields;
my %seen;
# Keep a copy of the arguments so we can output the data in
# the same order we read them.
my @files = @ARGV;
while (<>) {
chomp;
next unless /^#.*=/;
s/^#\s*//;
my ($key,$val) = split /\s*=\s*/, $_, 2;
if (!defined($seen{$key})) {
push @fields, $key;
$seen{$key} = 1;
};
# $ARGV is the name of the current file being read
# by the `while(<>)` loop.
$vars{$ARGV}{$key} = $val;
};
print join(",", @fields), "\n";
foreach my $f (@files) {
next unless -r $f; # skip output for filenames that weren't readable
print join(",", @{$vars{$f}}{@fields}), "\n";
};
The script keeps track of the order that files were read and that field names were seen because perl hashes are inherently un-ordered (this is common for most implementations of associative arrays in most languages). I could have written it to sort the keys during the output stage (perl has a very useful built-in sort
function), so at least they would be output in a predictable order, but I thought it better to use a few variables to remember the original order.
It works with any number of output fields and doesn't care what the field names or the values are. In a matching line, everything after any leading whitespace and before the first =
sign is the "key", and everything after the first =
sign is the value. Whitespace around the =
is not included in either key or value (the line is split on \s*=\s*
rather than just =
). see perldoc -f split
for details on the split function.
If a given key appears more than once in a file, then the value in the last occurrence will be the one that is output. If you want it to keep the first and ignore any subsequent occurrence(s), add the following line before the $vars{$ARGV}{$key} = $val;
line:
next if (defined($vars{$ARGV}{$key}));
Sample run:
$ chmod +x ./vars2csv.pl
$ ./vars2csv.pl file1 file2
var1,var 2,var3
1,2,3
a,b,c
Worth noting: this script ignores all lines that don't start with a #
and contain an =
. That means it processes ALL lines matching that criterion - including any comment lines that co-incidentally happen to contain an =
that you didn't intend to define a variable. Depending on what, exactly, is in your input files, this may be a bug that needs to be fixed (by figuring out either a pattern to exclude these unwanted lines, or by crafting a better pattern to match only wanted lines).
BTW, I added the next unless -r $f;
line in the script because I tested the script with filename args that didn't exist and with permissions that prevented them from being read. Perl printed a warning message when such errors occurred, but the script printed a line with empty fields separated by commas. This line prevents that output.
The script will also print a line of empty fields separated by commas for readable files that didn't contain any var=value
comments. If you want to prevent output for those files too, add the following before the print join...
line.
next unless (keys %{ $vars{$f} }); # skip output for files with NO key=val comments
Files with some but not all fields will be printed with correct values for the fields they do have and empty values for any missing fields. e.g. a file with only # var1=1
would print 1,,
as the output line. If you want to skip output for these files:
next unless (@{$vars{$f}}{@fields}); # skip output for files missing ANY key
file1
there arevar1
,var2
,var3
andvar4
then the same will be infile2
too?