I have a file (data.dat) which contains 3 columns. I want to read in two of them with the shell script and fill the contents into a template file. The file looks like this:

12 180 2390
14 177 2210
16 173 2130

So far I can do that with one variable (which is column 2 in the data-file) as follows:

for a in `cat data.dat | gawk '{print $2}'`; do
 sed s/A/$a/ daily.template daily.template > daily.inp

Now I want to read in column 3 as well and give the according value (b) of the same line as column 2 to the same template. The template contains variables called "A" and "B". I cannot do a nested for loop as I don't want to iterate each value of a over each value of b.

How can I achieve this?


First of all thanks for all your help!

Yes, it might be more easy to understand if I give more information about the daily.template file. It is looking similiar to:

dens_column O3 300.00
dens_column h20 B
sza A
day_of_year 172
output sum

A and B should be replaced in every loop over the lines. The first column in data.dat does not mean anything, it just displays the time of a measurement and this parameter is not needed in "daily.inp". I just want one "daily.inp" for each line where A and B are replaced with the according value of column 2 and 3 in the according line. So a daily.inp file like:

dens_column O3 300.00
dens_column h20 2390
sza 180
day_of_year 172
output sum

Then the next file separate:

dens_column O3 300.00
dens_column h20 2210
sza 177
day_of_year 172
output sum

And so on...

  • So daily.template contains the same number of lines as data.dat? And each line in daily.template has variables A and B? And each line in data.dat should be matched up with the same line in daily.template? Are both files in the expected order? Does the 12, 14, 16 have any meaning? Does it also appear in daily.template?
    – Mikel
    Sep 4, 2013 at 14:49
  • 2
    Please provide a sample daily.template and the expected output.
    – Mikel
    Sep 4, 2013 at 14:50
  • The resulting file will contain only one line (the last you worked on). If that's not wanted, move the >daily.inp after the done.
    – ott--
    Sep 4, 2013 at 15:38

3 Answers 3


Then that should only be a matter of:

while read discard x y discard; do
  sed "s/A/$x/g;s/B/$y/g" < daily.template > "daily.$x-$y.inp"
done < data.dat

The read command will read one line at a time from data.dat, put the first field (fields being blank separated, and where backslash can be used to escape the field or line separator) in the variable $discard, the second in $x, the third in $y and the rest of the fields in $discard.

  • Thanks. If I try this I receive the error: Syntax error: "done" unexpected (expecting "do"). And where in this command do I define where to find (which column) A and B ?
    – user46489
    Sep 4, 2013 at 16:27
  • @user46489, please make sure you copy it as-is, and make sure the pasted text has normal Unix-style line endings. Sep 4, 2013 at 16:31
  • Thanks for the explanation! Unfortunately I still receive the same error. I was copying the code and replacing A, B, data.dat with the according variables/file. I checked that there is no extra space somewhere. I don't know what else could be wrong...
    – user46489
    Sep 4, 2013 at 17:06
  • @user46489, if it says it's expecting do, that's because you've not entered the do as in the text I wrote. One possible explanation is that you entered do<Cr> (<Cr> being the invisible carriage return characters sometimes added at the end of the line when working on Microsoft operating systems). Which is why I suggested you make sure you write the file in Unix format. When editing the file with vim, type :set ff=unix Sep 4, 2013 at 22:22
  • thank you. Indeed I had a problem with the format as I am using Linux in a virtual box on my Windows computer. So now I solved the do problem. But, if I want to process following code: while read discard sza dens_h2o discard; do sed "s/SZA/$sza/g;s/DENS_H2O/$dens_h2o/g" < daily_SUR.template > daily_SUR.inp echo $sza $dens_h2o done < datenum_sza_dens_lowRes_test.dat I got the error message::not foundy_loop.test.sh: 4: :No such file or directory and then the correct echo of the variables...
    – user46489
    Sep 6, 2013 at 12:30

You can save the two columns in gawk printing $2 and $3. Then, extract those in separate variables a1 and a2 using col. From there on, it is straightforward to sed the two template variables.

  • Thanks @unxnut! Could you give me the complete command as I am not using shell script usually, so I have already problems following your description! That would be very kind! Thanks a lot!
    – user46489
    Sep 4, 2013 at 16:05

You can use termsql for that job:

while read a b; do
   termsql -xwi daily.template "update tbl set COL2='$a' where COL1='h20'; update tbl set COL1='$b' where COL0='sza'" > daily$a$b.inp
done < <(termsql -wi data.dat "select COL1,COL2 from tbl")

Advantage: If you know SQL, you don't have to learn new syntax.

You can get termsql here: https://gitorious.org/termsql

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .