the following Situation: I have about 2000 Files that are named like "dens_Run0_set0.#" where # is some Number and they are filled like this

       1   0 1.000000e-01  6.3043370127282594e-01 -9.0687990423248266e-03  255 9.4509489123595358e-10 
       1   1 1.000000e-01  6.4071465952974904e-01  2.1357878278801461e-02  254 9.6824903650372107e-10 
       1   2 1.000000e-01  6.6258330124264830e-01 -8.3797056064819717e-03  255 9.5136742928048079e-10 
       1   3 1.000000e-01  6.2986715737376164e-01 -8.3306463914899122e-03  254 9.8325203534699313e-10 
       1   4 1.000000e-01  6.2984096793040101e-01 -2.2874605754168442e-03  255 9.7199825965788023e-10 
       1   5 1.000000e-01  6.3544494921916339e-01  4.0850692190967192e-04  255 9.3920704546945899e-10 
       1   6 1.000000e-01  6.1993535905745978e-01  1.6905867851164658e-03  255 9.2219992753328484e-10 
       1   7 1.000000e-01  6.6403589120818651e-01 -7.7065063672989709e-03  255 9.5683005962854395e-10 
       1   8 1.000000e-01  6.6151286798685760e-01  2.2987801390020952e-02  255 9.5036933878134278e-10 
       1   9 1.000000e-01  6.4426718104580993e-01 -9.1829913570412227e-03  255 9.5504193442508267e-10 
       6   0 1.000000e-01  4.0147426676412579e-02 -1.0968071205388696e-01  281 9.5556063062218755e-10 
       6   1 1.000000e-01  3.1792618041921133e-01  1.5324342699326129e-01  281 9.4066654465763122e-10 
       6   2 1.000000e-01  2.0570344663186310e-01 -5.7959543075412512e-02  281 9.4459629007559442e-10 
       6   3 1.000000e-01  1.4426831089076586e-01 -8.0085259971383493e-02  281 9.3488350394466124e-10 
       6   4 1.000000e-01  1.2697447761062600e-01 -7.9212954607404143e-02  281 9.4211360934792765e-10 
       6   5 1.000000e-01  9.4787370490545683e-02 -1.0902733768125472e-01  281 9.6087293677271646e-10 
       6   6 1.000000e-01 -3.6135020945645614e-02 -6.3683751812277478e-02  281 9.6366270518899455e-10 
       6   7 1.000000e-01  1.8645702206170731e-01 -1.1584221723023802e-01  281 9.7691299494329087e-10 
       6   8 1.000000e-01  3.4541099054452690e-01  5.8025780529230055e-02  281 9.4433605379862229e-10 
       6   9 1.000000e-01  2.6448819749081548e-01 -1.3873411819319395e-01  281 9.3928487210348521e-10 

Now I want to merge all Lines of all 2000 Files, that start with a "1" into a new File and all that start with a "6" into a different File. How can I do this?

1 Answer 1


Using find and calling awk:

find . -maxdepth 1 -type f -name 'dens_Run0_set0.*' \
    -exec awk '{ print >>$1 ".output" }' {} +

The find command would find all the regular files in the current directory (only) with names satisfying the given pattern. For batches of these files, a very short awk program would be executed.

The awk program simply outputs each line in the found files to new files having names like 1.output or 6.output depending on the data in the first field.

It is assumed that the output files do not already exist (or the existing files would be appended to). It is further assumed that the first field always forms a valid safe filename (there are no checks to prevent overwriting existing files elsewhere in the filesystem).

Would you want to restrict the awk program to only handle lines with exactly the two numbers 1 or 6 in the first field (in case there are other lines with other numbers that you don't want to extract), then change the awk code to

$1 == 1 || $1 == 6 { print >>$1 ".output" }

Depending on your system, the following shorter command may also work. If it doesn't, you would get an "Argument list too long" due to there being too many files.

awk '{ print >$1 ".output" }' dens_Run0_set0.*

The find command above gets around this error by executing awk several times on batches of files (possibly not all files at once).


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .