2

I am testing S3 command line tool performances at the moment and I am testing them using files I have created with fallocate. Im measuring the time different tools take to upload, say, 200 files of 1MB each to the S3 storage. Is there a difference between the files I have created with fallocate compared to a different file I have created myself, for example a .txt document of 1MB size? Or is a test.txt file created with fallocate the exact same as a test.txt created with notepad?

I want to make sure that the tests I make are realistic and not just fast because fallocate creates a different type of file.

3

Depending on the tools used, and the filesystem used, there may be differences.

From the application point of view, fallocate created files are usually full of zeros; your measuring times will be skewed and they will be even more affected if you are using either protocols or filesystems that have compression or deduplication built-in, as the amount of data transferred/written won't ever reflect the same operations with an equivalent file of the same size created by other means.

Also as Stéphane Chazelas correctly points out, the files appear as containing 0s because the extents are marked as "unwritten". When reading those, the system returns 0s without even reading the disk.

It would be preferred either to use .txt files, or better yet files generated with random binary data.

  • 1
    Also, the files appear as containing 0s because the extents are marked as "unwritten". When reading those, the system returns 0s without even reading the disk. – Stéphane Chazelas May 3 '17 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.