1

I have a simulation model where I can run a simulation in multiple parts, where for each part output is generated with an increasing cycle number. Output files may look like the following:

a.000.nc   b.000.nc  c.000.nc
a.001.nc   b.001.nc  c.001.nc
a.002.nc   b.002.nc  c.002.nc

To concatenate these files, I use nco, more specifically the record concatenator ncrcat. I used to do this manually like the following for each file name (a through c):

ncrcat a.00{0..2}.nc a.nc 

How would I go about writing a bash script that performs the ncrcat command on all files and concatenates all files with increasing cycle number (so a.000.nc, a.001.nc and a.002.nc to a.nc; next file would be b.000.nc, b.001.nc, b.002.nc to b.nc)? I got as far with egrep, etc. to extract which cycle numbers are present in an ls output, but do not know how to extract the file name.

Thanks!

4
  • How would you determine "all parts that belong together"? Please edit your question to include this information May 23, 2020 at 10:19
  • I edited the post. So in the end all files with the same prefixes (a, b or c) should include all correspoding cycle numbers (a.00{0..2}.nc to a.nc, b.00{0..2}.nc to b.nc etc).
    – sfluck
    May 23, 2020 at 11:37
  • Do you know beforehand that you have a* and b* and c*, or are you saying that you need to discover this? May 23, 2020 at 12:04
  • Are those .nc files POSIX text files or something else that requires you to run a specialized command ncrcat on them?
    – Ed Morton
    May 25, 2020 at 17:26

1 Answer 1

1

Given a list of files like this (the ones from your example, plus all numbers from 000 to 100 for a):

$ ls
a.000.nc  a.007.nc  a.014.nc  a.021.nc  a.028.nc  a.035.nc  a.042.nc  a.049.nc  a.056.nc  a.063.nc  a.070.nc  a.077.nc  a.084.nc  a.091.nc  a.098.nc  c.001.nc
a.001.nc  a.008.nc  a.015.nc  a.022.nc  a.029.nc  a.036.nc  a.043.nc  a.050.nc  a.057.nc  a.064.nc  a.071.nc  a.078.nc  a.085.nc  a.092.nc  a.099.nc  c.002.nc
a.002.nc  a.009.nc  a.016.nc  a.023.nc  a.030.nc  a.037.nc  a.044.nc  a.051.nc  a.058.nc  a.065.nc  a.072.nc  a.079.nc  a.086.nc  a.093.nc  a.100.nc
a.003.nc  a.010.nc  a.017.nc  a.024.nc  a.031.nc  a.038.nc  a.045.nc  a.052.nc  a.059.nc  a.066.nc  a.073.nc  a.080.nc  a.087.nc  a.094.nc  b.000.nc
a.004.nc  a.011.nc  a.018.nc  a.025.nc  a.032.nc  a.039.nc  a.046.nc  a.053.nc  a.060.nc  a.067.nc  a.074.nc  a.081.nc  a.088.nc  a.095.nc  b.001.nc
a.005.nc  a.012.nc  a.019.nc  a.026.nc  a.033.nc  a.040.nc  a.047.nc  a.054.nc  a.061.nc  a.068.nc  a.075.nc  a.082.nc  a.089.nc  a.096.nc  b.002.nc
a.006.nc  a.013.nc  a.020.nc  a.027.nc  a.034.nc  a.041.nc  a.048.nc  a.055.nc  a.062.nc  a.069.nc  a.076.nc  a.083.nc  a.090.nc  a.097.nc  c.000.nc

You can do:

for f in *.nc; do echo ${f//\.*}; done | sort | uniq | 
    while read prefix; do echo ncrcat "$prefix".*nc "$prefix".nc; done

That produces this output:

ncrcat a.000.nc a.001.nc a.002.nc a.003.nc a.004.nc a.005.nc a.006.nc a.007.nc a.008.nc a.009.nc a.010.nc a.011.nc a.012.nc a.013.nc a.014.nc a.015.nc a.016.nc a.017.nc a.018.nc a.019.nc a.020.nc a.021.nc a.022.nc a.023.nc a.024.nc a.025.nc a.026.nc a.027.nc a.028.nc a.029.nc a.030.nc a.031.nc a.032.nc a.033.nc a.034.nc a.035.nc a.036.nc a.037.nc a.038.nc a.039.nc a.040.nc a.041.nc a.042.nc a.043.nc a.044.nc a.045.nc a.046.nc a.047.nc a.048.nc a.049.nc a.050.nc a.051.nc a.052.nc a.053.nc a.054.nc a.055.nc a.056.nc a.057.nc a.058.nc a.059.nc a.060.nc a.061.nc a.062.nc a.063.nc a.064.nc a.065.nc a.066.nc a.067.nc a.068.nc a.069.nc a.070.nc a.071.nc a.072.nc a.073.nc a.074.nc a.075.nc a.076.nc a.077.nc a.078.nc a.079.nc a.080.nc a.081.nc a.082.nc a.083.nc a.084.nc a.085.nc a.086.nc a.087.nc a.088.nc a.089.nc a.090.nc a.091.nc a.092.nc a.093.nc a.094.nc a.095.nc a.096.nc a.097.nc a.098.nc a.099.nc a.100.nc a.nc
ncrcat b.000.nc b.001.nc b.002.nc b.nc
ncrcat c.000.nc c.001.nc c.002.nc c.nc

Assuming that is correct, remove the echo and run the command again toa ctually concatenate the files.

Explanation

  • for f in *.nc; do ...; done: iterate over all files (and directories, if any are present) ending in .nc.
  • echo ${f//\.*};: remove everything after the first .. This will just print out the prefixes a, b and c.
  • sort | uniq: keep only one of each prefix.
  • while read prefix; do ...; done: iterate over each prefix.
  • ncrcat "$prefix".*nc "$prefix".nc: run nrcat on all filenames starting with this prefix and ending in .nc, giving $prefix.nc as the outpufile.
1
  • for f in *.nc; do echo ${f//\.*}; done | sort | uniq could be implemented without a loop as printf '%s\n' *.nc | cut -d. -f1 | sort -u and the subsequent loop using its output while read prefix; do echo ncrcat "$prefix".*nc "$prefix".nc; done could be implemented without a loop as xargs -n 1 -I {} sh -c 'echo ncrcat {}.*nc {}.nc'. With different input file names the versions without a loop would be more robust, in this case they'd just be faster.
    – Ed Morton
    May 25, 2020 at 17:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .