-1

I have a data list data.txt that look something like the following:

Sample A
   12 
   0
   29
   238
   0
   4

Basically it is a biological sample (single column) and the expression of certain genes (320+ rows). I'd like multiply each of the values on each row against each other, so that I get an output.txt something along the lines of this:

-      -    -      -    -   -
0      -    -      -    -   -
348    0    -      -    -   -
2856   0    82824  -    -   -
0      0    0      0    -   -
48     0    116    952  0   -

I'm currently just doing it manually on excel which is a pain (I'm a biologist and am not too good with UNIX or coding in general). Is there any way I could do this with awk (or some other command) instead?

Also, I actually have ~300 samples (i.e. 300 separate data.txt files), it would be even better if I could run one script for a data.txt file containing all 300 samples (i.e. 300 columns, 320 rows, and values for each column on each row) and get 300 separate output.txt files for each sample (column).

1

It's not actually that hard in awk, for example you could slurp the whole file into a (1d) array, and then do a double loop e.g. for a single data file:

awk '
  NR > 1 {
    x[NR-1] = $1;
  }
  END {
    for (i=1; ;i++) {
      if (!(i in x))
        break;
      for (j=1; ;j++) {
        if (!(j in x))
          break;
        if (j < i)
          printf "%-6d", x[j]*x[i];
        else
          printf "%-6s", "-";
      }
      printf "\n";
    }
  }
  ' data.txt

Note that if you don't insist on filling the upper triangle with dashes, you can process lines much more simply on the fly, i.e. without loading the whole file:

awk '
  NR > 1 {
    x[NR-1] = $1;
    for (j=1;j<NR-1;j++)
      printf "%-6d", $1*x[j];
    printf "%-6s\n", "-";
  }
  ' data.txt
-
0     -
348   0     -
2856  0     6902  -
0     0     0     0     -
48    0     116   952   0     -

Such an approach may provide a benefit (in terms of memory usage) if your datasets are large.

Processing multiple file arguments needs a bit of adjustment - if you have GNU awk you can make use of the BEGINFILE and ENDFILE rules e.g.

gawk '
  BEGINFILE {
    delete x;
  }
  FNR > 1 {
    x[FNR-1] = $1;
    next;
  }
  ENDFILE {
    n = length(x);
    for (i=1;i<=n;i++) {
      for (j=1;j<=n;j++) {
        if (j < i)
          printf "%-6d", x[j]*x[i];
        else
          printf "%-6s", "-";
      }
      printf "\n";
    }
    printf "\n";
  }
  ' data1.txt data2.txt

If you don't have gawk you can simulate the BEGINFILE/ENDFILE by checking when FNR rolls back to 1.


If your datasets are large however, I'd suggest using something that's built for matrix operations, such as GNU octave (basically you seem to want the lower triangle of the product, correct?)

You can either do it in an interactive octave interpreter, or in 'batch' mode like so:

$ octave -q << 'EOF'
> x = textread('data.txt', '%f', 'headerlines', 1);
> M = tril(x * x', -1);
> disp(M);
> EOF
      0      0      0      0      0      0
      0      0      0      0      0      0
    348      0      0      0      0      0
   2856      0   6902      0      0      0
      0      0      0      0      0      0
     48      0    116    952      0      0

Depending on your exact output requirements you can use the simple save or dlmwrite functions to write M to a file, or fprintf if you want really fine grained output format control. If you want the upper triangle zeros replaced by dashes, you will probably need to look at cell arrays.

-1

There is no single program which performs this specific task, so some (beginner level) coding is needed.

Awk is not particularly suitable as a programming language even if it might be possible to write an awk script for this task. Perl might be a better language for this job, but any general purpose programming or scripting language should do. As you're not into coding, you will probably have to find someone willing to write a program for you, and leave the choice of language to them (unless they're proposing something that requires disproportionate setup effort.)

As far as I know, answers on SE are not supposed to contain complete code for specific tasks, but if someone posts such code, you might prefer their answer :-)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.