The Amazon Linux versions use device mapper as the Docker storage driver. Device mapper uses layered copy-on-write block devices to the containers. When no data has been written, then it hardly uses any disk spaces. As pages in the file system are being written, then it will allocate data from the device mapper and it will start using diskspace.
The default disk size for device mapper is 10GB, so when running 6 images you might require 60GB if all pages are being written to. It's unlikely that all pages are being written to, but the default EC2 instance only has 8GB which is not enough.
When a file is written then it is written to several blocks. If the blocks haven't been written yet, then it will be allocated from the device pool. When the file is removed, then the block is marked as unused in the filesystem, but the pool doesn't know about it. Only when fstrim is ran, then the filesystem releases the pages from the device manager.
NGINX maintains a cache on /tmp/cache and it's constantly written to. If the filesystem keeps using different blocks, then it will keep allocating blocks until all 10GB is mapped to the device manager. There are several solutions to this problem:
- Use a device mapper device that is large enough to fit all the blocks.
- Use a smaller base size (search for dm.basesize) to make sure the images cannot reach 10GB.
- Switch to a distribution that uses AUFS storage.
You can also run fstrim regularly to make sure freed blocks are returned to the pool. This might be tricky, because during peak load it might not run often enough and the pool might still run out of disk space.