0

I have several folders each containing several .nc files. I want to loop through all folders and for each folder do some calculations (mergetime) using cdo on files that have similar pattern. So far I have written the following:

for dir in /mnt/meteo_a/africa_cordex/historical/0.44/pr/*/     
do 
 dir=${dir%*/}
 echo ${dir##*/} 
 export dir
 cd $dir
 pwd
  for f in `find . -type f -regex /(.*?\day)/*`
  cdo mergetiem io
  done
done

The files in each folder have names like the following:

pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_day_19500101-19501231.nc
pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_day_19510101-19551231.nc
pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1_day_19491201-19501231.nc
pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1_day_19510101-19551231.nc

I want to apply the mergetime CDO command on all files that have similar pattern before _day_.

I want to merge the first two files and the last two files. More specifically, merge files that start with

"pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_" 

and likewise merge files that start with

"pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1"

The output of the mergetime can be something like:

 {pattern}_mergetime.nc 
6
  • It is not clear which files you want to select with your find command. Please show a list of files with some matching and some non-matching files and state which ones you want to select or which files belong into the same group(s). What exactly means "similar pattern before _day_"?
    – Bodo
    Commented Feb 12, 2019 at 12:19
  • I want to merge the first two files and the last two files. More specifically, merge files that start with "pr_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_" and likewise merge files that start with "pr_AFR-44_ICHEC-EC-EARTH_historical_r12i1p1_CLMcom-CCLM4-8-17_v1". Commented Feb 12, 2019 at 12:20
  • Please add the clarification to your question instead of writing a comment.
    – Bodo
    Commented Feb 12, 2019 at 12:28
  • I had a quick look at the documentation of cdo, and I found out that the mergetime operator requires the name of the outfile. The required arguments are as follows: cdo mergetime one_or_more_input_files one_outfile. So, what or how do you want to name your output files?
    – Niko Gambt
    Commented Feb 12, 2019 at 13:16
  • It can be whatever is before "day" with a tag like: {pattern}_mergetime.nc Commented Feb 12, 2019 at 13:18

1 Answer 1

1

This script snippet would work with your example files

for file in *.nc
do 
    echo "$file"
done | sed 's/_day_.*//' | sort -u | while read -r pattern
do
    cdo mergetime "${pattern}"* "${pattern}_mergetime.nc"
done

Explanation:

  • The for loop with echo prints one file name per line.
  • The sed command removes _day_ and all following characters.
  • sort -u sorts the partial file names and removes duplicates.
  • while read -r pattern reads one pattern per line and loops over the patterns
  • "${pattern}"* is expanded by the shell to all file names starting with the pattern

Better than the for loop with echo might be

find . -maxdepth 1 -type f -name '*.nc'

This will print all file names matching *.nc in the current directory without subdirectories.

You can combine this with a loop over all subdirectories similar to the script in the question

for dir in /mnt/meteo_a/africa_cordex/historical/0.44/pr/*/     
do 
    dir=${dir%*/}
    echo ${dir##*/} 
    export dir
    pushd $dir
    pwd

    find . -maxdepth 1 -type f -name '*.nc' | sed 's/_day_.*//' | sort -u | while read -r pattern
    do
        cdo mergetime "${pattern}"* "${pattern}_mergetime.nc"
    done

    popd
done

Instead of cd I propose pushd to allow going back with popd later.

You could also replace the for loop over the directories with an additional find

find /mnt/meteo_a/africa_cordex/historical/0.44/pr -maxdepth 1 -mindepth 1 -type d | while read dir
do
    pushd "$dir"

    find . -maxdepth 1 -type f -name '*.nc' | sed 's/_day_.*//' | sort -u | while read -r pattern
    do
        cdo mergetime "${pattern}"* "${pattern}_mergetime.nc"
    done

    popd
done
1
  • @MariaKarypidou If it works you can accept the answer. I read in a comment that the cdo command needs an output file argument. I will add this to my answer.
    – Bodo
    Commented Feb 12, 2019 at 13:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .