8

I'm on a Linux busybox 1.27 only system so no output=progress available, no busybox's own implementation of pv which is pipe_progress nor pv itself.

I have two questions. The first is based on https://www.linux.com/training-tutorials/show-progress-when-using-dd/. It says that by sending the USR1 signal to dd it "pauses" the process and dd after printing its current status will continue with the job it was doing. I'm trying to do some benchmark tests with dd so I would like to have minimal impact on the dd operation. I want to get an output of the current operation every second because the data that's passing through dd is fluctuating and it is important to me to recognize when the transfer rate drops.

First question: Is it true that 'dd' "pauses" every time it receives a USR1 signal?

If dd pauses every second then I'll be adding hours to the operation when tens of gigabytes are being transferred.

Second question: Assuming yes as an answer to the first question, I would like to know if it's possible to get dd to print its current status without sending any signal to the process, maybe some kind of redirection for STDOUT (like 2>&1)?

What I'm referring to is:

# bs with 1Mib so I can have more control on the test.
dd if=/dev/zero of=/dev/null bs=1048576 count=1024

# Printing current operation status.
sudo kill -USR1 $dd_pid 
4
  • 4
    Have you tried to measure the delay? Something like time dd … without and with while sleep 1; do kill -s USR1 $(pidof dd); done running aside. It may be the delay turns out to be acceptable to you. Nov 2, 2021 at 7:18
  • 4
    What kind of device over what bus are you dding? Typically the CPU usage is not the limiting factor for dd (watch top while dd is in progress!), so while "pausing" the process for some microseconds the transfer will probably continue to write data from the cache via DMA and have no influence at all on the transfer speed.
    – Philippos
    Nov 2, 2021 at 7:51
  • @KamilMaciorowski Thank you. I'll be checking that out but I'm a little bit hesitant into doing time due to cache/buffering happening. I'm not quite sure how the mechanisms behind the scene are going to interfere when measuring with time but is worth trying to get an idea. Hopefully it will not make much of a difference, a minimum delay is ok.
    – abacox
    Nov 2, 2021 at 7:59
  • @Philippos I'm trying to set up a reliable method with different embed boards running a bare linux system, Android phones over USB (with and without chroot) and computer's NIC 10/100 (one way or another I was able to connect them all). So consistency is very important, as far as I can stick with standard practices/tools I should be fine. All of them over TCP directly or TCP tunneling (TCP over different protocols like RNDIS, not "real" TCP). I really don't want to start transferring several gigabytes of data without any expectation of what might happen. But I have hope on cache/buffer.
    – abacox
    Nov 2, 2021 at 8:08

2 Answers 2

9

dd if=/dev/zero of=/dev/null bs=1048576 count=1024

Note that dd can mangle data, at least when using the bs parameter. And its performance advantage is small at best if you hand-pick an optimal block size for your particular system configuration: cat or cp can do better, and at most is only slightly slower. So don't use dd if you don't have to.

Note that since version 1.23, BusyBox uses the sendfile system call to copy data, instead of using read and write. Only plain copies such as cat and cp use sendfile, however: dd is forced to use read/write because it wants precise control over the sizes. So with BusyBox ≥1.23, cat and cp are very likely to be faster than dd.

Is it true that 'dd' "pauses" every time it receives an USR1 signal?

Technically yes, it has to “pause” to handle the signal. However the pause is only a few CPU instructions (the most costly part by far is printing the progress output). So this doesn't invalidate your benchmark in any way.

If dd pauses every second then I'll be adding hours to the operation when tens of gigabytes are being transferred.

No, you have your orders of magnitude wrong. You'll be adding maybe .1% of the time on a single-CPU thread. The main cost is kernel time for the benchmarking program, not dd, so it's intrinsic in what you want to do, not in the way you implement it.

if its possible to get dd to print its current status without sending any signal to the process

Well, no. There's already a simple, historically established, standard, easy way to do it. Why would there be another way which would be harder to implement?


On Linux, there is a generic way to know the point that a copy has reached. It doesn't depend on what program is doing the copy, though it doesn't always work with special files. Find out the process ID $pid that's doing the copy, and which file descriptors it's using for input and output. dd reads from fd 0 and writes to fd 1. BusyBox cp typically reads from fd 3 and writes to fd 4. You can check which file is open on which file descriptor through the symbolic links in /proc/$pid/fd.

$ cp /dev/zero /dev/null & pid=$!
$ readlink /proc/$pid/fd/3
/dev/zero
$ readlink /proc/$pid/fd/4
/dev/null

And you can check the position on the file descriptor $n$ in /proc/$pid/fd/$n.

$ cat /proc/$pid/fdinfo/4
pos:    74252288
flags:  0100001
mnt_id: 27

However, note that the file descriptor position may not updated with special files such as /dev/zero, /dev/null, pipes or sockets. It's always updated with regular files. I don't know if it's updated for block devices. So it likely won't give you any information for copying between /dev/zero and /dev/null, but it might work in your real use case.

2
  • 1
    Yes. In fact the effective overhead may actually be zero, because, in the situation described, it sounds like dd is mostly I/O bound - so it can afford to spend a few CPU cycles sending a message, and input/output buffering will mean that the operation isn't delayed as a result.
    – psmears
    Nov 2, 2021 at 15:34
  • It's not the bs parameter that you need to worry about, instead it is the count parameter. Outside of using count, the only time I've found that a partial read can cause problems is with tape drives - iflag=fullblock is useful in that case
    – Dark
    Nov 3, 2021 at 0:21
8

You can also query the progress via the /proc interface.

# dd bs=1M if=/dev/mmcblk0 of=/dev/null &
# pidof dd
1358

So the information about this process is to be found in /proc/1358:

# ls -l /proc/1358/fd
total 0
lr-x------    1 root     root            64 Nov  2 09:16 0 -> /dev/mmcblk0
l-wx------    1 root     root            64 Nov  2 09:16 1 -> /dev/null
lrwx------    1 root     root            64 Nov  2 09:16 2 -> /dev/pts/0

Filehandle 0 is the if=/dev/mmcblk0, now where is the progress at?

# cat /proc/1358/fdinfo/0
pos:    2132803584
flags:  0400000
mnt_id: 17
# cat /proc/1358/fdinfo/0
pos:    2366636032
flags:  0400000
mnt_id: 17
# cat /proc/1358/fdinfo/0
pos:    2587885568
flags:  0400000
mnt_id: 17

In this fashion, for busybox dd, you can also derive progress from its fdinfo pos value.

That said, sending the USR1 signal in moderation should have minimal impact on performance. On an embedded system with few resources, polling the fdinfo might just as well have a similar impact.

1
  • 1
    With a modern busybox, cat and cp are faster than dd because they use sendfile rather than read/write. But fdinfo might not be updated — I don't know if it's updated for block devices. Nov 2, 2021 at 8:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .