1

I'm running a MariaDB 10.1.34 instance on AWS RDS. Instance specs are as follows:

  • Instance Class: db.t2.large
  • Storage: 100GiB gp2
  • DB Size: 1.2 GiB

I'm trying to create a mysqdump from an EC2 instance located in the same availability zone as the db. While this works, the performance is extremely bad. It takes around 15 minutes to dump the database ( which is only 1.2 GiB in size ) with mysqldump.

I've been monitoring disk and network usage on the ec2 instance. In the first 5 seconds of mysqldump, network usage spikes around 600 Mbit/s and then dropping to only around 200 kbit/s. Same with the disk usage.

Strangely, rebooting the Database instance seems to fix the behaviour temporarily. mysqldump finished in 30 seconds one time, and became slow again afterwards.

The Write IOPS on the DB instance hover around 10 - 15, Read iops only exceeded 20 after restarting the instance.

I'd be grateful for any advice on how to debug this issue further.

EDIT:

I should also add my mysqldump command:

ubuntu@server:~$ mysqldump --version
mysqldump  Ver 10.13 Distrib 5.7.23, for Linux (x86_64)
ubuntu@server:~$ mysqldump -h <db_host> -p -P 3306 -u <read_only_user> --quick --single-transaction --skip-lock-tables <database> > dump.sql 

closed as too broad by Rui F Ribeiro, JigglyNaga, Romeo Ninov, user88036, G-Man Oct 13 '18 at 2:33

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

1

This is a classic case where you've been hit by T2/T3 burstable performance.

As you may know when T2/T3 instances are idle they accumulate "CPU credits" that they then use for running fast when needed. However these credits deplete when the instance is running at full speed and when there are none left the instance slows down. See here for more details: CPU Credits and Baseline Performance for Burstable Performance Instances

Although the above only talks about CPU credits from experience I know it directly affects network performance as well, although I wasn't able to find any document in that regard from Amazon. And that's what you observed too.

To work around it you've got 2 options:

  1. Switch to M4 / M5 instance type - these have constant performance and are not affected by CPU credits. They are slightly more expensive than T2 / T3 thuogh.

  2. Enable T2 unlimited setting (see here) that will top-up your instance credits whenever they drop to 0 for some extra cost. It may be the best option for your use case. However note that running T2 unlimited at full speed all the time is more expensive than running M4 instance of the same size.

Not the answer you're looking for? Browse other questions tagged or ask your own question.