0

I have several folders each containing several .nc files. I want to loop through all folders and for each folder do some calculations (mergetime) using cdo on files that have similar pattern. So far I have written the following:

for dir in /mnt/meteo_a/africa_cordex/historical/0.44/pr/*/     
do 
 dir=${dir%*/}
 echo ${dir##*/} 
 export dir
 cd $dir
 pwd
  for f in `find . -type f -regex /(.*?\day)/*`
  cdo mergetiem io
  done
done

The files in each folder have names like the following:

pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_day_19500101-19501231.nc
pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_day_19510101-19551231.nc
pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1_day_19491201-19501231.nc
pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1_day_19510101-19551231.nc

I want to apply the mergetime CDO command on all files that have similar pattern before _day_.

I want to merge the first two files and the last two files. More specifically, merge files that start with

"pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_" 

and likewise merge files that start with

"pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1"

The output of the mergetime can be something like:

 {pattern}_mergetime.nc 
  • It is not clear which files you want to select with your find command. Please show a list of files with some matching and some non-matching files and state which ones you want to select or which files belong into the same group(s). What exactly means "similar pattern before _day_"? – Bodo Feb 12 at 12:19
  • I want to merge the first two files and the last two files. More specifically, merge files that start with "pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_" and likewise merge files that start with "pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1". – Maria Karypidou Feb 12 at 12:20
  • Please add the clarification to your question instead of writing a comment. – Bodo Feb 12 at 12:28
  • I did that, thank you! – Maria Karypidou Feb 12 at 12:33
  • I had a quick look at the documentation of cdo, and I found out that the mergetime operator requires the name of the outfile. The required arguments are as follows: cdo mergetime one_or_more_input_files one_outfile. So, what or how do you want to name your output files? – Niko Gambt Feb 12 at 13:16
0

This script snippet would work with your example files

for file in *.nc
do 
    echo "$file"
done | sed 's/_day_.*//' | sort -u | while read -r pattern
do
    cdo mergetime "${pattern}"* "${pattern}_mergetime.nc"
done

Explanation:

  • The for loop with echo prints one file name per line.
  • The sed command removes _day_ and all following characters.
  • sort -u sorts the partial file names and removes duplicates.
  • while read -r pattern reads one pattern per line and loops over the patterns
  • "${pattern}"* is expanded by the shell to all file names starting with the pattern

Better than the for loop with echo might be

find . -maxdepth 1 -type f -name '*.nc'

This will print all file names matching *.nc in the current directory without subdirectories.

You can combine this with a loop over all subdirectories similar to the script in the question

for dir in /mnt/meteo_a/africa_cordex/historical/0.44/pr/*/     
do 
    dir=${dir%*/}
    echo ${dir##*/} 
    export dir
    pushd $dir
    pwd

    find . -maxdepth 1 -type f -name '*.nc' | sed 's/_day_.*//' | sort -u | while read -r pattern
    do
        cdo mergetime "${pattern}"* "${pattern}_mergetime.nc"
    done

    popd
done

Instead of cd I propose pushd to allow going back with popd later.

You could also replace the for loop over the directories with an additional find

find /mnt/meteo_a/africa_cordex/historical/0.44/pr -maxdepth 1 -mindepth 1 -type d | while read dir
do
    pushd "$dir"

    find . -maxdepth 1 -type f -name '*.nc' | sed 's/_day_.*//' | sort -u | while read -r pattern
    do
        cdo mergetime "${pattern}"* "${pattern}_mergetime.nc"
    done

    popd
done
  • This works! Thank you! – Maria Karypidou Feb 12 at 13:33
  • @MariaKarypidou If it works you can accept the answer. I read in a comment that the cdo command needs an output file argument. I will add this to my answer. – Bodo Feb 12 at 13:37

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.