1

I have a project on AWS which will eventually have over 100 client databases each estimated in the 1~2GB size. I need to do daily backups of each database via mysqldump not the standard AWS RDS daily snapshots.

In bash I loop through a list and dump each db one after the other which is likely to take a few hours. My question is

Is it adviseable to do a bash fork and dump the DBs in parallel?

  • To give a meaningful opinion it would be useful to know where its the filesystem you are dumping the DBs – Rui F Ribeiro Dec 19 '18 at 7:07
  • the plan is to dump to an EFS(NFS) then move to S3 standard bucket. – user60434 Dec 19 '18 at 9:04
1

Assuming you're having a file client_db.list that contains a list of databases name, all databases belong to one RDS instance live-db.example.internal

> cat client_db.list
client01_dbname
client02_dbname
client03_dbname
...
...
client_template

Basic solution

The solution is using GNU parallel apt install parallel for executing mysqldump each database in parallel

parallel -j2 -a client_db.list mysqldump -uroot -pXXXXXXXXX -hlive-db.example.internal \
                                         --single-transaction \
                                         --skip-lock-tables {} '>' ./backup/{.}.sql

Details

-j2 means executing mysqldump for 2 databases at the same time, based on number of vCPU of RDS instance

The output will be:

ls -lh
total 3.4G
-rw-r--r-- 1 jenkins jenkins  26M Jun  4 10:56 client01_dbname.sql
-rw-r--r-- 1 jenkins jenkins  26M Jun  4 10:54 client02_dbname.sql
-rw-r--r-- 1 jenkins jenkins  26M Jun  4 10:54 client03_dbname.sql
...
...
-rw-r--r-- 1 jenkins jenkins  26M Jun  4 10:54 client_template.sql

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.