1

I use GNU Parallel along a for loop like this:

for BAND in $(seq 1 "$BANDS") ;do
   # Do not extract, unscale and merge if the scaled map exists already!
   SCALED_MAP="era5_and_land_${VARIABLE}_${YEAR}_band_${BAND}_merged_scaled.nc"
   MERGED_MAP="era5_and_land_${VARIABLE}_${YEAR}_band_${BAND}_merged.nc"
   if [ ! -f "${SCALED_MAP+set}" ] ;then
       echo "log $LOG_FILE Action=Merge, Output=$MERGED_MAP, Pixel size=$OUTPUT_PIXEL_SIZE_X $OUTPUT_PIXEL_SIZE_Y, Timestamp=$(timestamp)"
       echo "gdalmerge_and_clean $VARIABLE $YEAR $BAND $OUTPUT_PIXEL_SIZE_X $OUTPUT_PIXEL_SIZE_Y"
   else
       echo "warning "Scaled map "$SCALED_MAP" exists already! Skipping merging.-""
   fi
done |parallel -j 20 --joblog "parallel.${JOB_CLUSTER_PROCESS}.log"
log "$LOG_FILE" "Action=Merge, End=$(timestamp)"

(for the records: where ${JOB_CLUSTER_PROCESS}" a variable given by HTCondor).

In the logs I see only entries of the first command

echo "log $LOG_FILE Action=Merge, Output=$MERGED_MAP, Pixel >size=$OUTPUT_PIXEL_SIZE_X $OUTPUT_PIXEL_SIZE_Y, Timestamp=$(timestamp)"

which is actually a custom way to log actions during this loop in the following way:

# tell what you are doing
function log {
    echo "${@: 2}" 2>&1 >> "$1" ;
}
export -f log

Is it possible to get the second line

echo "gdalmerge_and_clean $VARIABLE $YEAR $BAND $OUTPUT_PIXEL_SIZE_X >$OUTPUT_PIXEL_SIZE_Y"

only or along with the first together included in the .log file created by --joblog?

Improve this question
10
  • I'm not sure how your code works (the parallel section, idw what the log section (at the end of your script) does ) But if you want only this line: echo "gdalmerge_and_clean $VARIABLE $YEAR $BAND $OUTPUT_PIXEL_SIZE_X >$OUTPUT_PIXEL_SIZE_Y" I think you can use:
    – Edgar Magallon
    Dec 2, 2022 at 16:20
  • Redirect the first line to stderr: echo "log $LOG_FILE Action=Merge, Output=$MERGED_MAP, Pixel size=$OUTPUT_PIXEL_SIZE_X $OUTPUT_PIXEL_SIZE_Y, Timestamp=$(timestamp)" >&2
    – Edgar Magallon
    Dec 2, 2022 at 16:22
  • And after done (the end of the for loop) redirect stderr to.null or some file: done > /dev/null or done > /path/to/somefile
    – Edgar Magallon
    Dec 2, 2022 at 16:24
  • Thank you for the idea @EdgarMagallon. log is just a custom function that writes whatever is there in a custom .log file. I am exploring the use of --joblog and I realised that --joblog records only the first line.
    – Nikos Alexandris
    Dec 2, 2022 at 22:18
  • Maybe it's a bit difficult to achieve what you want. (or maybe not, but given that I cannot understand the script completely, not sure how all is working). However, I think about piping while read -r line; do after done, like this: done | while read -r line; do parallel -j 20 --joblog "parallel.${JOB_CLUSTER_PROCESS}.log" ; done
    – Edgar Magallon
    Dec 3, 2022 at 4:41

1 Answer 1

Reset to default
2

--joblog only adds to the joblog when the job is finished.

You are giving GNU Parallel two jobs:

log ...
gdalmerge_and_clean ...

log finishes fast and is added to joblog, but gdalmerge_and_clean probably takes longer to run.

I think you should consider rewriting your job as a function and call that:

doit() {
   BAND=$1
   # Do not extract, unscale and merge if the scaled map exists already!
   SCALED_MAP="era5_and_land_${VARIABLE}_${YEAR}_band_${BAND}_merged_scaled.nc"
   MERGED_MAP="era5_and_land_${VARIABLE}_${YEAR}_band_${BAND}_merged.nc"
   if [ ! -f "${SCALED_MAP+set}" ] ;then
       log $LOG_FILE Action=Merge, Output=$MERGED_MAP, Pixel size=$OUTPUT_PIXEL_SIZE_X $OUTPUT_PIXEL_SIZE_Y, Timestamp=$(timestamp)
       gdalmerge_and_clean $VARIABLE $YEAR $BAND $OUTPUT_PIXEL_SIZE_X $OUTPUT_PIXEL_SIZE_Y
   else
       warning "Scaled map "$SCALED_MAP" exists already! Skipping merging.-"
   fi
}
export -f doit

seq 1 "$BANDS" |
  parallel -j 20 --joblog "parallel.${JOB_CLUSTER_PROCESS}.log" doit {}
log "$LOG_FILE" "Action=Merge, End=$(timestamp)"

I recommend you try --dry-run if GNU Parallel does something you do not expect. It will tell you what commands it intends to run.

I think it will be time well spent if you read chapter 1+2 of GNU Parallel 2018 (https://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html or download it at: https://doi.org/10.5281/zenodo.1146014)

It should take you less than 20 minutes, and your command line will love you for it.

Improve this answer
1
  • Indeed, I did this before reading this answer : put both in one function! Thank you for the explanations and recommendation. I am reading the suggested pages. Have read and did some of the exercises in the past. However, it's always useful to revisit the manuals. I completely forgot, for example the --keep-order! Mea culpa.-
    – Nikos Alexandris
    Dec 8, 2022 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.