Arithmetic between 2 files generating a series of new files

Question

I have a tab separated model input file I would like to vary for an ensemble analysis formatted similar to this

input.txt

/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.5
abies_grandis 2.5 0.4
larix_occidentalis 1.5 0.3

I have another file of multipliers with one per line such as this

multipliers.txt

0.5
0.6
0.7

I would like to generate a series of new input files where one field (wsg) is multiplied by a single multiplier from the second file. In this example there would be 3 new files corresponding to the 3 multipliers. The output files would look like the following:

file1.txt (wsg * 0.5)

/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.25
abies_grandis 2.5 0.2
larix_occidentalis 1.5 0.15

file2.txt (wsg * 0.6)

/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.3
abies_grandis 2.5 0.24
larix_occidentalis 1.5 0.18

file3.txt (wsg * 0.7)

/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.35
abies_grandis 2.5 0.28
larix_occidentalis 1.5 0.21

This seems like something that could be done using awk and a for loop, but my knowledge of awk is not at the level necessary to execute this task. How should I approach this?

You're basically talking about matrix multiplication. I would probably do this in R. — cryptarch, Feb 12 at 19:51
How many lines max could there be in multipliers.txt? The answer greatly affects how to implement a solution. — Ed Morton, Feb 12 at 22:27
OK then every solution that uses a shell loop to call any tool multiple times will be very slow and any solution that only uses awk but doesn't close the output files as it goes will fail unless you use GNU awk. — Ed Morton, Feb 13 at 13:41
@EdMorton I'm sure you're right about non-GNU erroring out for too many files, but you also mentioned that GNU awk would get slow. I just tested with a version that closes the file (albeit after every write, which may explain this, see my answer) and 1001 multipliers and the version closing the opened files was a bit slower. Are you sure gawk would experience the slowdown you mention? — terdon♦, Feb 13 at 16:09

Ed Morton · Accepted Answer · 2021-02-13 17:09:22Z

Using any awk in any shell on every Unix box no matter now many multipliers you have:

$ ls *.txt
input.txt  multipliers.txt

$ cat tst.awk
NR==FNR {
    if ( pastHdr ) {
        ++numLines
        wsg[numLines] = $NF
        sub(/[[:space:]][^[:space:]]+$/,"")
        rest[numLines] = $0
    }
    else {
        hdr = hdr $0 ORS
        if ( $1 == "***" ) {
            pastHdr = 1
        }
    }
    next
}
{
    out = "file" FNR ".txt"
    printf "%s", hdr > out
    for (lineNr=1; lineNr<=numLines; lineNr++) {
        print rest[lineNr], wsg[lineNr] * $0 > out
    }
    close(out)
}

$ awk -f tst.awk input.txt multipliers.txt

$ ls *.txt
file1.txt  file2.txt  file3.txt  input.txt  multipliers.txt

$ head file*.txt
==> file1.txt <==
/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.25
abies_grandis 2.5 0.2
larix_occidentalis 1.5 0.15

==> file2.txt <==
/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.3
abies_grandis 2.5 0.24
larix_occidentalis 1.5 0.18

==> file3.txt <==
/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.35
abies_grandis 2.5 0.28
larix_occidentalis 1.5 0.21

The above is very similar to the solution that @guest_7 posted, I posted it because theirs:

Uses many single character variable names which IMHO make a script of this size much harder for newcomers to understand,
Use NF-- to delete the wsg value but decrementing NF is undefined behavior and so non-portable, and
Hard-codes how many lines the header is (which may not be wrong, it's just my preference to determine that from analyzing the text - but that might also be wrong depending on whether or not the real input always looks like the posted example and, if not, in what way it differs).

Thanks for this! I've realized my sample was not specific enough, though. This seems to be working on the last element of each row, but what would be better would be the third element of each row. Is there an easy change that could be made? — sethparker, Feb 15 at 17:09
Probably but ask a new followup question that has concise, testable sample input and expected output that this solution doesn't work for to make sure we understand the problem. — Ed Morton, Feb 15 at 17:37
I have posted a followup here unix.stackexchange.com/questions/634648/… Thanks again — sethparker, Feb 15 at 20:25

Kusalananda · Accepted Answer · 2021-02-12 20:24:15Z

Assuming that the multiplication needs to happen to exactly the last whitespace-delimited fields on lines 6 onward in input.txt. Using a simple shell loop:

count=0
while IFS= read -r mult; do
    count=$(( count + 1 ))
    awk -v mult="$mult" 'NR >= 6 { $NF *= mult }; 1' <input.txt >"file$count.txt"
done <multipliers.txt

This reads a multiplier from multipliers.txt in each iteration of the loop, and then applies that to the value of the last field ($NF) on lines 6 onward (NR >= 6). Each line, whether modified or not, is then printed (this is what the trailing 1 does).

The output filename is created via the static prefix file, the counter count, and the static suffix .txt.

You could get rid of the multipliers.txt file if you wish (if you always had a small number of unchanging multipliers):

count=0
for mult in 0.5 0.6 0.7; do
    count=$(( count + 1 ))
    awk -v mult="$mult" 'NR >= 6 { $NF *= mult }; 1' <input.txt >"file$count.txt"
done

guest_7 · Accepted Answer · 2021-02-13 16:51:10Z

We read the input.txt file into an awk array since they are just 6 lines.

awk '
  NR==FNR {
    if (NR<6) {h=h sep $0}
    else {
      n++;    b[n]=$NF
      $NF=""; a[n]=$0
    }
    sep = ORS
    next
  }
  {
    close(f); f = fp($1)
    print h > f
    for (i=1; i<=n; i++) {
      print a[i] b[i]*$1 > f
    }
  }
  function fp(m) {
    sub(/[.]/, "p", m)
    return "file@mult=" m ".txt"
  }
' input.txt multipliers.txt

Results:

.
├── file@mult=0p5.txt
├── file@mult=0p6.txt
├── file@mult=0p7.txt
├── input.txt
└── multipliers.txt

::::::::::::::
file@mult=0p5.txt
::::::::::::::
/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.25
abies_grandis 2.5 0.2
larix_occidentalis 1.5 0.15

::::::::::::::
file@mult=0p6.txt
::::::::::::::
/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.3
abies_grandis 2.5 0.24
larix_occidentalis 1.5 0.18

::::::::::::::
file@mult=0p7.txt
::::::::::::::
/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.35
abies_grandis 2.5 0.28
larix_occidentalis 1.5 0.21

This is a nice approach, but why would you needlessly complicate things by using a filename with special characters? — terdon♦, Feb 13 at 16:21
Thanks @terdon. The reason for filenames being like they are is I wanted the multiplier info in it and a p for decimal point in place of a literal dot coz they interfere with basename / dirname determination. Of course, YMMV. — guest_7, Feb 13 at 16:49

Ed Morton · Accepted Answer · 2021-02-13 17:19:57Z

Here's one way (GNU awk):

gawk '{ 
        if(NR==FNR){
            mlt[$1]
        }
        else{ 
            oldVal=$3 
            for(m in mlt){ 
                outFile="outfile_"m".txt"; 
                if(a){  $3 = oldVal * m} 
                if($4 == "wsg"){  a=1  } 
                print $0 > outFile
            }
        }
    }' multipliers.txt input.txt

That produces the following files:

outfile_0.5.txt:

/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.25
abies_grandis 2.5 0.2
larix_occidentalis 1.5 0.15

outfile_0.6.txt:

/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name 0 wsg
abies_lasiocarpa 2 0.3
abies_grandis 2.5 0.24
larix_occidentalis 1.5 0.18

outfile_0.7.txt:

/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name 0 wsg
abies_lasiocarpa 2 0.35
abies_grandis 2.5 0.28
larix_occidentalis 1.5 0.21

Explanation

if(NR==FNR) { mlt[$1] }: if we are reading the first file, save the value in the mlt array.
else { ... }: if we are not reading the first file.
oldVal=$3 : save the original value of the third field so we can later multiply it. Note that this also saves the values of lines we don't want to multiply or even those that don't have a third field, but it doesn't matter because we won't use it for those lines.
for(m in mlt){ ... }: iterate over the mlt array (the multipliers), saving each multiplier as m.
outFile="outfile_"m".txt";: set the output file name to be outfile_ then the current multiplier m and then a .txt extension since you seem to be using those.
if(a){ $3 = oldVal * m}: if the variable a is set, multiply the third field by m.
if($4 == "wsg"){ a=1 } : if the 4th field of this line is wsg, set a to 1. This is used in the step mentioned above to know if we should multiply.
print $0 > outFile: print the current line to the output file. The current line has already been modified by multiplying its 3rd field.

Apparently, non-GNU awks require that you close the opened files and I have also been told that even GNU awk may become slower if you don't. If you encounter this sort of issue, you can try this version which closes the file after every read. If you use this, because it will append to the output file, make sure you delete any already existing output files before running.

awk '{ 
        if(NR==FNR){
            mlt[$1]
        }
        else{ 
            oldVal=$3 
            for(m in mlt){ 
                outFile="outfile_"m".txt"; 
                if(a){  $3 = oldVal * m} 
                if($4 == "wsg"){  a=1  } 
                print $0 >> outFile
                close(outFile)
            }
        }
    }' multipliers.txt input.txt

I tested both of the above solutions using a multiplier file with 1001 different values, and the second which closes the file was actually slower (0m0.131s) than the first which didn't close (0m0.107s). However, this might be because I am reopening and closing the file after each write which is not an efficient approach at all. For context, Ed Morton's solution which also closes the files cleanly, but is clever enough not to do it after every write, only took 0m0.058s on the same test data, so faster than both my solutions.

There should be a close somewhere. Otherwise, for more multipliers it cud error out — guest_7, Feb 13 at 0:53
Thanks @guest_7, answer edited. I admit I was only thinking of small datasets like what the OP described (1000 multipliers and tiny files), but you make a good point regardless. — terdon♦, Feb 13 at 16:16
Not closing the files and letting gawk handle opening and closing them as needed is expected to be faster than opening/closing each output file on every write to it, it's just not as portable. Both will be slower than opening/closing each output file once per output file (as opposed to once per line written to each output file). — Ed Morton, Feb 13 at 17:14

terdon · Accepted Answer · 2021-02-13 16:32:00Z

j=1; while read i; do awk -v i="$i" 'NR>5{$NF=$NF*i}1' input.txt >file$j.txt\(wsg*"$i"\);j=$(($j+1)); done < multipliers.txt

output

@praveen:~$ cat file1.txt\(wsg\*0.5\) 
/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.25
abies_grandis 2.5 0.2
larix_occidentalis 1.5 0.15

===========
praveen@praveen:~$ cat file2.txt\(wsg\*0.6\) 
/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.3
abies_grandis 2.5 0.24
larix_occidentalis 1.5 0.18

========

praveen@praveen:~$ cat file3.txt\(wsg\*0.7\) 
/* Preciptation in mm */
10 30 40 50 23

### Species description
*** sp_name LMA wsg
abies_lasiocarpa 2 0.35
abies_grandis 2.5 0.28
larix_occidentalis 1.5 0.21

Python

#!/usr/bin/python
mu=open('Multipliers.txt','r')
ik=open('input.txt','r')
t=ik.readlines()
d=1
for g in mu:
    er=open("file{0}.txt(wsg*{1})".format(d,g.strip()),'w')
    for h in range(0,len(t),1):
        if (int(h) < 5):
            print t[h].strip()
            er.write(t[h].strip())
            er.write('\n')
        else:
            k=t[h].split(' ')
            co=float(k[-1].strip())*float(g)
            print " ".join(k[0:-1])+" "+str(co)
            er.write(" ".join(k[0:-1])+" "+str(co))
            er.write('\n')
    d=d+1

willus · Accepted Answer · 2021-02-13 16:43:54Z

Why use a convenient, interpreted shell language or python when you can do everything in C?? ;)

gcc -o gomult gomult.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static void procfile(char *inputfile,double m);


int main(int argc,char *argv[])

    {
    char inputfile[256],multfile[256];
    FILE *in,*mult;
    char buf[256];

    if (argc<3)
        {
        printf("usage: gomult <infile> <multfile>\n");
        return(10);
        }
    strncpy(inputfile,argv[1],255);
    strncpy(multfile,argv[2],255);
    in=fopen(inputfile,"r");
    if (in==NULL)
        {
        printf("Cannot open input file %s for input.\n",inputfile);
        return(20);
        }
    fclose(in);
    mult=fopen(multfile,"r");
    if (mult==NULL)
        {
        printf("Cannot open multiplier file %s for input.\n",multfile);
        return(30);
        }
    /* Get next line from multiplier file */
    while (fgets(buf,255,mult)!=NULL)
        {
        int na;
        double m;

        /* If it's a number, process input file with it */
        na=sscanf(buf,"%lf",&m);
        if (na==1)
            procfile(inputfile,m);
        }
    fclose(mult);
    return(0);
    }


static void procfile(char *inputfile,double m)

    {
    FILE *in;
    FILE *out;
    char outfile[256];
    char buf[256];
    static int count=1;
    int cc;

    in=fopen(inputfile,"r");
    if (in==NULL)
        {
        printf("Could not open input file %s for input(!).\n",inputfile);
        exit(10);
        }
    sprintf(outfile,"file%d.txt",count);
    count++;
    out=fopen(outfile,"w");
    if (out==NULL)
        {
        printf("Could not open output file %s for output.\n",outfile);
        return;
        }
    /* Fixed line count */
    cc=0;
    /* Get next line from input file */
    while (fgets(buf,255,in)!=NULL)
        {
        int na5,na;
        char name[256];
        double n1,n2;

        /* If line has string and two numbers, fix it, otherwise pass it through */
        /* na5 checks line for 5 numbers--need to pass that one through */
        na5=sscanf(buf,"%*f %*f %*f %lf %lf",&n1,&n2);
        na=sscanf(buf,"%s %lf %lf",name,&n1,&n2);
        if (na5==2 || na!=3)
            fprintf(out,"%s",buf);
        else
            {
            fprintf(out,"%s\t%g\t%g\n",name,n1,n2*m);
            cc++;
            }
        }
    fclose(out);
    fclose(in);
    printf("File %s written with %d changed lines.\n",outfile,cc);
    }

Stack Exchange Network

current community

your communities

more stack exchange communities

Arithmetic between 2 files generating a series of new files

6 Answers 6

Explanation

Your Answer

Not the answer you're looking for? Browse other questions tagged shell-script awk arithmetic or ask your own question.

Linked

Hot Network Questions

Arithmetic between 2 files generating a series of new files

6 Answers 6

Explanation

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged shell-script awk arithmetic or ask your own question.

Linked

Related

Hot Network Questions