I have a gridded model data (separated by different depth) saved in text files. The structure is like this:

Format of every column in a row in the text file:

x_coordinate y_coordinate density

There are about 400*400 points in every text file(as a plane in certain depth).



Here, ### is a number represents the depth(in Z direction). The number can be an integer or fraction. For example, now I have these files: "dep0", "dep0.5", "dep10", "dep300", that means those data are 400*400 gridded data in x-y plane form for depth of 0, 0.5, 10 and 300.

Now, I would like to pick up the number in the filename (aka the depth) and add it to the third column of each row, combine all of them together. Also the depth should be sorted from minima to maxima. So the output file should looks like this(for example):

x_coordinate y_coordinate z_coordinate density
0            0            0            2.5
0            1            0            2.5
...          ...          0            2.6
400          400          0            2.9
0            0            0.5          2.8
...          ...          0.5          2.9
0            0            10           3.2
...          ...          10           3.3
...          ...          300          4.7
...          ...          300          4.8

At first I was doing it with this script:

 if [ -f "xyp/dep"$i ];then
  awk '{print $1, $2,'$i',$3}' "xyp/dep"$i >> "xyzp/area1"

Then I figure out it would miss any files which depth is not integer, since the variable $i in the for loop increase by 1 in every round.

I have tried to use sed and find -exec, but I kept getting errors. The difficulty for me is that I don't quite understand how to use $, '', <<< correctly to redirect or pipe the value to awk or other function. Please help me with this problem.


I have come up with this script:

depnumbers=$(ls xyp | sed -e 's/dep//g' |sort -n)
filecount=$(ls xyp | wc -l)

  dep=$(awk '{print $'$i'}' <<< $depnumbers)
  awk '{print $1, $2,'$dep',$3}' "xyp/dep"$dep >> "xyzp/area1"

It works just fine. Is there any way to simplify or improve this script for such task? Actually I'm new to bash and still think something is wrong...not sure

  • Please edit your post and add an example of a filename with a fraction. – garethTheRed Oct 15 '14 at 10:17
  • Please edit your question and add an example of the actual data and your desired output. Isn't the order of the rows in the second file important? We can't understand exactly what you need without an example. – terdon Oct 15 '14 at 11:26
  • Sorry for the misspell, wrong grammar and unclear statements. Tell me if there are still something not clear. – Pakox.Wang Oct 15 '14 at 12:26
  • What is the name of the file awk is reading ? – user78605 Oct 15 '14 at 13:08
  • The accepted answer to this question may help you tidy your loop :-) – garethTheRed Oct 15 '14 at 13:08

For only adding numbers into the file.

awk 'NR>1{print $1,$2,substr(FILENAME,7),$4 }' xyp/dep* > "xyzp/area1"

For sorting by numbers.

ls -1v xyp/dep* | xargs awk 'NR>1{print $1,$2,substr(FILENAME,7),$4 }' > "xyzp/area1"

For sorting from minus numbers.

 ls xyp/dep* | sort -t 'p' -k 3 -n | xargs awk 'NR>1{print $1,$2,substr(FILENAME,7),$4 }' > "xyzp/area1"

This would regard the character p as separator and make the function sort -n only affect on the following numbers.

  • Because the number that follows the filename doesn't have same digits. If I have files at depth of 1, 3, 10, 15, 20, 300, the reading priority will be like 1=>10=>15=>20=>3=>300, since the substr only compares the number from specific position. Also it should be substr(FILENAME,7) because the FILENAME contain path xyp/. Still many thanks :) I'll have to read some manual. – Pakox.Wang Oct 15 '14 at 14:08
  • @Pakox.Wang Yeah sorry, i didn't test and forgot it included the directory. Will sorting the new file on the third field not work ? – user78605 Oct 15 '14 at 14:13
  • @Pakox.Wang Check update – user78605 Oct 15 '14 at 14:25
  • Sorry I just went home...will test tomorrow. Checked the manual and it seems fine... but I want to ask if it's still okay when we have the depth number in negative? Just asking, currently have no such issue but I'm aware of it since the data is from Geology model. BTW, I think we should put a wildcard at ls -1v xyp/dep* – Pakox.Wang Oct 15 '14 at 14:59
  • Yeah i should have,thanks :) anyway i added another answer for minus/negative numbers – user78605 Oct 15 '14 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.