3 Grammar, wording
source | link

I'm going to reframe your question to highlight some of the context:

Why is

fio --name=seqwrite --rw=write --bs=128k --numjobs=1 --size=16G --runtime=120 --iodepth=1 --group_reporting

faster than

time dd if=/dev/random of=./test_file bs=128k count=131072 

on a FreeBSD 11.2 system with 768GB RAM, SAS HDDs and ZFS configured as a RAIDZ2 with compression enabled?

A major difference is that fio is pre-making the file before doing the timing tests against it:

seqwrite: Laying out IO file (1 file / 16384MiB)

whereas dd is likely doing file extending writes (which will cause metadata updates). Also you have so much RAM (768G) and you're writing so little data in comparison to it (16G) there's a strong chance your writes can be held in RAM (and not actually written to disk until much later). This is likely in the fio case where the file has been premade and very little file metadata needs to be modified per I/O. You can at least tell fio not to say it's done until all written data to be written back from the kernel at the end of the job by using end_fsync=1.

(NB: There's a subtle hint that I/O is being buffered when you see completion latencies much lower than what you know your disk can do:

clat (usec): min=28, max=2585, avg=48.03, stdev=24.04

Can your spinning disk really complete an I/O in 28 microseconds? If not it likely got buffered somewhere)

Finally, fio defaults to reusing the same the pattern in subsequent blocks. Since there's compression going on this could further distort on thenimprove your fio throughput (but this will be dependent on things like the ZFS recordsize). To check this, tell fio to make its buffers incompressible (which in turn turns refill_buffers on) and see if the throughput drops (which it did in your case).

TLDR; the fio and dd commands you gave are not testing the same thing. You need to be aware of things like whether your files already exist at the correct size, how compressible the data you are writing is and whether you are accounting for things like kernel buffering by writing too little data and not checking whether it's all been written back to disk.

I'm going to reframe your question to highlight some of the context:

Why is

fio --name=seqwrite --rw=write --bs=128k --numjobs=1 --size=16G --runtime=120 --iodepth=1 --group_reporting

faster than

time dd if=/dev/random of=./test_file bs=128k count=131072 

on a FreeBSD 11.2 system with 768GB RAM, SAS HDDs and ZFS configured as a RAIDZ2 with compression enabled?

A major difference is that fio is pre-making the file before doing the timing tests against it:

seqwrite: Laying out IO file (1 file / 16384MiB)

whereas dd is likely doing file extending writes (which will cause metadata updates). Also you have so much RAM (768G) and you're writing so little data in comparison to it (16G) there's a strong chance your writes can be held in RAM (and not actually written to disk until much later). This is likely in the fio case where the file has been premade and very little file metadata needs to be modified per I/O. You can at least tell fio not to say it's done until all written data to be written back from the kernel at the end of the job by using end_fsync=1.

(NB: There's a subtle hint that I/O is being buffered when you see completion latencies much lower than what you know your disk can do:

clat (usec): min=28, max=2585, avg=48.03, stdev=24.04

Can your spinning disk really complete an I/O in 28 microseconds? If not it likely got buffered somewhere)

Finally fio defaults to reusing the same the pattern in subsequent blocks. Since there's compression going this could further distort on then your fio throughput (but this will be dependent on things like the ZFS recordsize). To check this tell fio to make its buffers incompressible (which in turn turns refill_buffers on) and see if the throughput drops.

TLDR; the fio and dd commands you gave are not testing the same thing. You need to be aware of things like whether your files already exist at the correct size, how compressible the data you are writing is and whether you are accounting for things like kernel buffering by writing too little data and not checking whether it's all been written back to disk.

I'm going to reframe your question to highlight some of the context:

Why is

fio --name=seqwrite --rw=write --bs=128k --numjobs=1 --size=16G --runtime=120 --iodepth=1 --group_reporting

faster than

time dd if=/dev/random of=./test_file bs=128k count=131072 

on a FreeBSD 11.2 system with 768GB RAM, SAS HDDs and ZFS configured as a RAIDZ2 with compression enabled?

A major difference is that fio is pre-making the file before doing the timing tests against it:

seqwrite: Laying out IO file (1 file / 16384MiB)

whereas dd is likely doing file extending writes (which will cause metadata updates). Also you have so much RAM (768G) and you're writing so little data in comparison to it (16G) there's a strong chance your writes can be held in RAM (and not actually written to disk until much later). This is likely in the fio case where the file has been premade and very little file metadata needs to be modified per I/O. You can at least tell fio not to say it's done until all written data to be written back from the kernel at the end of the job by using end_fsync=1.

(NB: There's a subtle hint that I/O is being buffered when you see completion latencies much lower than what you know your disk can do:

clat (usec): min=28, max=2585, avg=48.03, stdev=24.04

Can your spinning disk really complete an I/O in 28 microseconds? If not it likely got buffered somewhere)

Finally, fio defaults to reusing the same the pattern in subsequent blocks. Since there's compression going on this could further improve your fio throughput (but this will be dependent on things like the ZFS recordsize). To check this, tell fio to make its buffers incompressible (which in turn turns refill_buffers on) and see if the throughput drops (which it did in your case).

TLDR; the fio and dd commands you gave are not testing the same thing. You need to be aware of things like whether your files already exist at the correct size, how compressible the data you are writing is and whether you are accounting for things like kernel buffering by writing too little data and not checking whether it's all been written back to disk.

2 Add more speculation and reframing of the answers and a summary
source | link

I'm going to reframe your question to highlight some of the context:

Why is

fio --name=seqwrite --rw=write --bs=128k --numjobs=1 --size=16G --runtime=120 --iodepth=1 --group_reporting

faster than

time dd if=/dev/random of=./test_file bs=128k count=131072 

on a FreeBSD 11.2 system with 768GB RAM, SAS HDDs and ZFS configured as a RAIDZ2 with compression enabled?

A major difference is that fio is pre-making the file before doing the timing tests against it:

seqwrite: Laying out IO file (1 file / 16384MiB)

whereas dd is likely doing file extending writes (which will cause metadata updates). Also you have so much RAM (768G) and you're writing so little data in comparison to it (16G) there's a strong chance your writes can be held in RAM (and not actually written to disk until much later). This is likely in the fio case where the file has been premade and very little file metadata needs to be modified per I/O. You can at least tell fio not to say it's done until all written data to be written back from the kernel at the end of the job by using end_fsync=1.

(NB: There's a subtle hint that I/O is being buffered when you see completion latencies much lower than what you know your disk can do:

clat (usec): min=28, max=2585, avg=48.03, stdev=24.04

Can your spinning disk really complete an I/O in 28 microseconds? If not it likely got buffered somewhere)

Finally By default fio will reusefio defaults to reusing the same the pattern in subsequent blocks. IfSince there's compression / deduplication going this could further distort on then your fio throughput could look artificially high and would drop if you(but this will be dependent on things like the ZFS recordsize). To check this tell fio to try and make it'sits buffers uncompressibleincompressible (which in turn turns refill_buffers on) and see if the throughput drops.

TLDR; the fio and dd commands you gave are not testing the same thing. You need to be aware of things like whether your files already exist at the correct size, how compressible the data you are writing is and whether you are accounting for things like kernel buffering by writing too little data and not checking whether it's all been written back to disk.

By default fio will reuse the same the pattern in subsequent blocks. If there's compression / deduplication going on then your throughput could look artificially high and would drop if you tell fio to try and make it's buffers uncompressible (which in turn turns refill_buffers on).

I'm going to reframe your question to highlight some of the context:

Why is

fio --name=seqwrite --rw=write --bs=128k --numjobs=1 --size=16G --runtime=120 --iodepth=1 --group_reporting

faster than

time dd if=/dev/random of=./test_file bs=128k count=131072 

on a FreeBSD 11.2 system with 768GB RAM, SAS HDDs and ZFS configured as a RAIDZ2 with compression enabled?

A major difference is that fio is pre-making the file before doing the timing tests against it:

seqwrite: Laying out IO file (1 file / 16384MiB)

whereas dd is likely doing file extending writes (which will cause metadata updates). Also you have so much RAM (768G) and you're writing so little data in comparison to it (16G) there's a strong chance your writes can be held in RAM (and not actually written to disk until much later). This is likely in the fio case where the file has been premade and very little file metadata needs to be modified per I/O. You can at least tell fio not to say it's done until all written data to be written back from the kernel at the end of the job by using end_fsync=1.

(NB: There's a subtle hint that I/O is being buffered when you see completion latencies much lower than what you know your disk can do:

clat (usec): min=28, max=2585, avg=48.03, stdev=24.04

Can your spinning disk really complete an I/O in 28 microseconds? If not it likely got buffered somewhere)

Finally fio defaults to reusing the same the pattern in subsequent blocks. Since there's compression going this could further distort on then your fio throughput (but this will be dependent on things like the ZFS recordsize). To check this tell fio to make its buffers incompressible (which in turn turns refill_buffers on) and see if the throughput drops.

TLDR; the fio and dd commands you gave are not testing the same thing. You need to be aware of things like whether your files already exist at the correct size, how compressible the data you are writing is and whether you are accounting for things like kernel buffering by writing too little data and not checking whether it's all been written back to disk.

1
source | link

By default fio will reuse the same the pattern in subsequent blocks. If there's compression / deduplication going on then your throughput could look artificially high and would drop if you tell fio to try and make it's buffers uncompressible (which in turn turns refill_buffers on).