Run Rsync for multiple files in parallel from 2 different directories and find elapsed time for each file and all files

Question

I have lots of dump files in 2 different directories that need to be uploaded using rsync, I want to run rsync for each dump file simultaneously to save time.

Dump Files:

Dir1:

-  /u05/expdpdump/exppdb/dir1/NoTDE_CDB_FULL1_01.dmp
-  /u05/expdpdump/exppdb/dir1/NoTDE_CDB_FULL1_02.dmp
-  /u05/expdpdump/exppdb/dir1/NoTDE_CDB_FULL1_03.dmp
-  /u05/expdpdump/exppdb/dir1/NoTDE_CDB_FULL1_04.dmp

Dir2:

-  /u05/expdpdump/exppdb/dir2/NoTDE_CDB_FULL2_01.dmp
-  /u05/expdpdump/exppdb/dir2/NoTDE_CDB_FULL2_02.dmp
-  /u05/expdpdump/exppdb/dir2/NoTDE_CDB_FULL2_03.dmp
-  /u05/expdpdump/exppdb/dir2/NoTDE_CDB_FULL2_04.dmp

Following rsync Command to run for each dumpfile from each directory in the background:

rclone sync /u05/expdpdump/exppdb/NoTDE_CDB_FULL_01.dmp NoTDE_Mig1:IC_dbbackup_config_datapump_xxxxxx

I managed to get the desired output to run rsync for each file, however, i want to track the start and end time for each file as well as overall elapsed time for the rsync jobs running in background. using time in for loop will provide only elapsed time for each file, however i also need overall elapsed time of Start and End Time after completing all jobs

dumpdir1="/u05/expdpdump/exppdb/dir1"

for i in $dumpdir1/*.dmp
do

    echo " time rclone sync  $i NoTDE_Mig1:IC_dbbackup_config_datapump_v00rcfh_iad3p2 &"
done

dumpdir2="/u05/expdpdump/exppdb/dir2"

for i in $dumpdir2/*.dmp
do
    echo " time rclone sync $i NoTDE_Mig1:IC_dbbackup_config_datapump_v00rcfh_iad3p2 &"
done```

Answer 1

Something like this:

# `time` will give the total time
time parallel -j0 --joblog mylog rclone sync {} NoTDE_Mig1:IC_dbbackup_config_datapump_v00rcfh_iad3p2 ::: /u05/expdpdump/exppdb/dir{1,2}/*.dmp
# the log contains the time per file
cat mylog 

Depending on your disks you can get better performance if you do not run all commands in parallel. You limit the number of jobs running in parallel to 5 by replacing -j0 with -j5.