6

I recently learned about the convenient "shared memory" file system at /dev/shm. I wanted to see if I could use this to speed up a sometimes-disk-bound program that writes to and reads from a directory, so I ran a little experiment on an EC2 instance (c4.8xlarge running Ubuntu 16.04):

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt
real    0m21.381s
$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt
real    0m20.266s
$ time yes asdfjkl | head -1000000000 > /dev/null
real    0m14.334s

The EC2 instance seems to have a write throughput to /dev/shm/, that is comparable to an EBS drive, which was surprising. htop indicates that the machine isn't using swap space in order to write to /dev/shm. The noticeably faster write to /dev/null in the third case indicates that I'm probably not bound by some other factor (e.g. CPU via the implementation of yes) in the first two cases.

I ran the same experiment on my personal computer -- enough memory that /dev/shm can hold 7.5 GB of asdfjkl\n by default, I can dig up more hardware details if anyone thinks they'll matter -- also running Ubuntu 16.04:

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt
real    0m36.520s
$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt
real    0m12.516s
$ time yes asdfjkl | head -1000000000 > /dev/null
real    0m11.252s

This is much closer to what I expected. Read throughput (writing to /dev/null) on both machines and from both file systems is roughly proportional to write throughput in the corresponding case.

Two other observations, which I don't know quite how to interpret:

  • On the EC2 instance, htop indicates memory usage comparable to the size of /dev/shm/fake.txt once it's written, while on my desktop, it does not.
  • On the EC2 instance, concurrent disk write congestion appears to slow down a write to shared memory by an amount comparable to how much the writes to disk were slowed down, while on my desktop, it does not.
4
  • 2
    You really need to ask Amazon. They won't tell you, but probably nobody else can. Commented Jan 24, 2018 at 5:36
  • 2
    You would improve the question by adding details of the swap usage for both systems to it.
    – JdeBP
    Commented Jan 24, 2018 at 6:29
  • 1
    /dev/zero might (?) be a better data source. Interesting question. Commented Jan 24, 2018 at 13:11
  • @JdeBP neither machine is using swap, clarified in the question. Thanks
    – James
    Commented Jan 24, 2018 at 13:40

1 Answer 1

0

If I had to guess, I'd say that the underlying volume is actually EBS-backed. It depends on the AMI what kind of root volume gets created EBS-backed or instance store backed.

And then it also depends what /dev/shm points too. I recently learned that there are usually 2 disks on an EC2 instance. You can see which one is which by running: sudo nvme list

See also: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html and https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/add-instance-store-volumes.html

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .