1

I have four files each contain daily station data.

  • file 1: prec
  • file 2: minT
  • file 3: maxT
  • file 4: wind

Here prec, minT, maxT and wind are files that store values of prec, minT, maxT and wind for stations 1 through n.

I want to store the data for each station like:

for station 1: prec minT maxT wind 
    station 2: prec minT maxT wind
    .
    .
    .
    .
    station n: prec minT maxT wind

EDIT #1

My four data fies look like as follows:

Prec

1        2        3        4        5        6        7        8
0        0        0        0        0        0        0        0
0.254        0.254        0.254        0.254        0.254        0.254        0.254        0.254
0        0        0        0        0        0        0        0

Tmin

1          2          3          4          5          6          7          8
-23.349        -23.339        -23.327        -23.316        -23.303        -23.291        -23.278        -23.266
-23.682        -23.683        -23.685        -23.687        -23.689        -23.692        -23.695        -23.698
-24.302        -24.301        -24.3        -24.299        -24.299        -24.299        -24.3        -24.302

Tmax

 1         2         3         4         5         6         7         8
-17.087        -17.082        -17.077        -17.072        -17.066        -17.06        -17.053        -17.046
-20.082        -20.095        -20.109        -20.124        -20.14        -20.157        -20.174        -20.191
-20.48        -20.481        -20.483        -20.485        -20.486        -20.488        -20.489        -20.49

wind

1        2        3        4        5        6        7        8
0        0        0        0        0        0        0        0
1.778        1.778        1.778        1.778        1.778        1.778        1.778        1.778
1.652        1.653        1.654        1.654        1.655        1.656        1.657        1.658

I want to format the data file for each point it's named with (that is for points 1 to 8) as below:

1

0        -23.349        -17.087        0
0.254        -23.682        -20.082        0
0        -24.302        -20.48        1.778

2

0        -23.339        -17.082        0
0.254      -23.683        -20.095        0
0        -24.301        -20.481        1.778

...and so on... to n.

Where the columns in each file would be: nprec, nTmin, nTmax, and nwind.

  • 2
    I think the question is not very clear. Please supply examples of the contents of each of the 4 files and what you want the final result to look like. – Joseph R. Aug 22 '13 at 1:00
  • Can you add a sample row or 2 from each of those files? Do they include the date the telemetry data was acquired? – slm Aug 22 '13 at 1:23
  • 1
    Please double check what I've tried to clean up in your question still makes sense. – slm Aug 22 '13 at 1:42
1

I think you could do this with paste, and possibly join.

paste file1 file2 > temp1
paste temp1 file3 > temp2
paste tmpe2 file4 > final

That should work if the files contain station 1 thru n data in order. If not, you will need to get the files aligned somehow. If the "stattion n" notation is part of every file, you should sort them:

sort -o file1 file1
sort -o file2 file2
... and so forth

Then, you could do join instead of paste as above. join has some tricky options about putting lines in output that don't have a match in the other file, so you may have to read the join man page very carefully and experiment a bit to get what you want.

| improve this answer | |
0

From your description it sounds like you'd like to iterate through the 4 files, Prec, Tmin, Tmax, and wind and take the column data from each and merge it into a singe file per each column.

The approach

One way to achieve this is by using a Bash for loop, paste and awk. The approach would go something like this:

for i in `seq 8`; do
    ... do stuff ...
done

The primary purpose of this loop would be to step through the columns, 1 through 8.

Pulling out columns

The next bit we need to put together is the ability to pull data from a file, column-wise. One idea would be to use awk to do this.

$ awk '{print $1}' Prec
1
0
0.254
0

We can take this a step further and get it to skip the first line, which is the reference number of the column, like so:

$ awk 'NR>1{print $1}' Prec
0
0.254
0

We can go even further by parameterizing the column we're printing by making it a variable that we pass into awk like so:

$ awk -v a=1 'NR>1{print $a}' wind
0
1.778
1.652
$ awk -v a=2 'NR>1{print $a}' wind
0
1.778
1.653

Puling out columns from multiple files

This bit is probably the least elegant bit to this solution, but I'm an awk noob, so when in doubt, go with a hack.

Here were using the output from multiple awk's which pull the column-wise data from each file, and pasting them together using paste.

Here's 2 of the files, just so it's easier to see.

column #1

$ paste <(awk -v a=1 'NR>1 {print $a}' Prec) <(awk -v a=1 'NR>1 {print $a}' Tmin)
0   -23.349
0.254   -23.682
0   -24.302

column #2

$ paste <(awk -v a=2 'NR>1 {print $a}' Prec) <(awk -v a=2 'NR>1 {print $a}' Tmin)
0   -23.339
0.254   -23.683
0   -24.301

To get columns from all 4 files we'll simply expand on this approach.

set our index ($i) to 1.

$ i=1
$ paste <(awk -v a=$i 'NR>1 {print $a}' Prec) <(awk -v a=$i 'NR>1 {print $a}' Tmin) <(awk -v a=$i 'NR>1{print $a}' Tmax) <(awk -v a=$i 'NR>1 {print $a}' wind)
0   -23.349 -17.087 0
0.254   -23.682 -20.082 1.778
0   -24.302 -20.48  1.652

set our index ($i) to 2.

$ i=2
$ paste <(awk -v a=$i 'NR>1 {print $a}' Prec) <(awk -v a=$i 'NR>1 {print $a}' Tmin) <(awk -v a=$i 'NR>1{print $a}' Tmax) <(awk -v a=$i 'NR>1 {print $a}' wind)
0   -23.339 -17.082 0
0.254   -23.683 -20.095 1.778
0   -24.301 -20.481 1.653

Putting it all together

So now let's incorporate the paste <(awk ..) <(awk ..) <(awk ..) <(awk ..) into the for loop.

$ for i in `seq 8`; do 
  echo "## $i ##"
  paste <(awk -v a=$i 'NR>1 {print $a}' Prec) <(awk -v a=$i 'NR>1 {print $a}' Tmin) <(awk -v a=$i 'NR>1{print $a}' Tmax) <(awk -v a=$i 'NR>1 {print $a}' wind)
done
## 1 ##
0   -23.349 -17.087 0
0.254   -23.682 -20.082 1.778
0   -24.302 -20.48  1.652
## 2 ##
0   -23.339 -17.082 0
0.254   -23.683 -20.095 1.778
0   -24.301 -20.481 1.653
## 3 ##
0   -23.327 -17.077 0
0.254   -23.685 -20.109 1.778
0   -24.3   -20.483 1.654
...

I've added an extra echo above so that in the output it's easier to see which column is being printed.

Writing everything to files

By adding this bit after the paste ... command, we can write the results out to the individual files as you specified.

paste ... | tee out$i.txt

The whole thing as a one liner:

$ for i in `seq 8`;do echo "## $i ##"; paste <(awk -v a=$i 'NR>1 {print $a}' Prec) <(awk -v a=$i 'NR>1 {print $a}' Tmin) <(awk -v a=$i 'NR>1{print $a}' Tmax) <(awk -v a=$i 'NR>1 {print $a}' wind) | tee out$i.txt;done

Which results in these files being written:

$ ll
total 48
-rw-rw-r-- 1 saml saml  71 Aug 21 23:17 out1.txt
-rw-rw-r-- 1 saml saml  72 Aug 21 23:17 out2.txt
-rw-rw-r-- 1 saml saml  70 Aug 21 23:17 out3.txt
-rw-rw-r-- 1 saml saml  72 Aug 21 23:17 out4.txt
-rw-rw-r-- 1 saml saml  71 Aug 21 23:17 out5.txt
-rw-rw-r-- 1 saml saml  71 Aug 21 23:17 out6.txt
-rw-rw-r-- 1 saml saml  70 Aug 21 23:17 out7.txt
-rw-rw-r-- 1 saml saml  71 Aug 21 23:17 out8.txt
| improve this answer | |
  • Thank you all for helping me. Problem solved successfully. – AiB Aug 22 '13 at 5:04
  • 1
    @user45611 - wonderful. If you think this answer was helpful and/or solved your issue directly, please consider voting it up and accepting it. – slm Aug 22 '13 at 5:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.