Loop through many folders and do calculations with files with similar pattern in bash

Question

I have several folders each containing several .nc files. I want to loop through all folders and for each folder do some calculations (mergetime) using cdo on files that have similar pattern. So far I have written the following:

for dir in /mnt/meteo_a/africa_cordex/historical/0.44/pr/*/     
do 
 dir=${dir%*/}
 echo ${dir##*/} 
 export dir
 cd $dir
 pwd
  for f in `find . -type f -regex /(.*?\day)/*`
  cdo mergetiem io
  done
done

The files in each folder have names like the following:

pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_day_19500101-19501231.nc
pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_day_19510101-19551231.nc
pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1_day_19491201-19501231.nc
pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1_day_19510101-19551231.nc

I want to apply the mergetime CDO command on all files that have similar pattern before _day_.

I want to merge the first two files and the last two files. More specifically, merge files that start with

"pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_" 

and likewise merge files that start with

"pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1"

The output of the mergetime can be something like:

 {pattern}_mergetime.nc 

Answer 1

This script snippet would work with your example files

for file in *.nc
do 
    echo "$file"
done | sed 's/_day_.*//' | sort -u | while read -r pattern
do
    cdo mergetime "${pattern}"* "${pattern}_mergetime.nc"
done

Explanation:

• The for loop with echo prints one file name per line.
• The sed command removes _day_ and all following characters.
• sort -u sorts the partial file names and removes duplicates.
• while read -r pattern reads one pattern per line and loops over the patterns
• "${pattern}"* is expanded by the shell to all file names starting with the pattern

Better than the for loop with echo might be

find . -maxdepth 1 -type f -name '*.nc'

This will print all file names matching *.nc in the current directory without subdirectories.

You can combine this with a loop over all subdirectories similar to the script in the question

for dir in /mnt/meteo_a/africa_cordex/historical/0.44/pr/*/     
do 
    dir=${dir%*/}
    echo ${dir##*/} 
    export dir
    pushd $dir
    pwd

    find . -maxdepth 1 -type f -name '*.nc' | sed 's/_day_.*//' | sort -u | while read -r pattern
    do
        cdo mergetime "${pattern}"* "${pattern}_mergetime.nc"
    done

    popd
done

Instead of cd I propose pushd to allow going back with popd later.

You could also replace the for loop over the directories with an additional find

find /mnt/meteo_a/africa_cordex/historical/0.44/pr -maxdepth 1 -mindepth 1 -type d | while read dir
do
    pushd "$dir"

    find . -maxdepth 1 -type f -name '*.nc' | sed 's/_day_.*//' | sort -u | while read -r pattern
    do
        cdo mergetime "${pattern}"* "${pattern}_mergetime.nc"
    done

    popd
done