0

Progress and estimated time to write without and with sync

I have found no tool (or straightforward method) that will include flushing the buffers when showing the progress and estimating the estimated time for the whole write process, ETA (Estimated Time of Arrival).

  • pv can show the time for the progress as seen by the operating system, but if the target drive is slow and there is a lot of RAM, it shows only the time until the data are written to a buffer. This time can be a small fraction of the the real time until the buffers are flushed.

  • dd writes a final report about amount of data used time and transfer rate. It can also be made to write 'progress' reports. It used to give a much better estimate than pv, but nowadays the USB drives and memory cards are still very slow, while the other processes are fast and the available memory for buffers big. So dd will also finish long before the buffers are flushed.

  • I can 'time' the write process including sync with the time command

    time ( write command; sync )
    

    and it will give me the real time used which is useful, but only after it has finished. It does not show the progress and does not estimate the total remaining time.

  • I can run iotop to show read and write processes and how fast things are read and written, but it does not estimate the remaining time.

How to show progress and estimated time for the whole write process?

How can I show progress and estimated time for the whole write process, ETA (Estimated Time of Arrival), including flushing the buffers with sync?

Link to related question

4
  • 1
    You could install and use dcfldd instead of using dd. It works exactly like dd on its paremeters but with the addition that it has a progress bar while the writing command is executing... – Rafael Muynarsk Nov 26 '19 at 18:23
  • 1
    I believe you could read (periodically) the dirty / writeback (don't know which one at this very moment), here's what I'm watching the flushing process: alias sync-watch='watch grep -e Dirty: -e Writeback: /proc/meminfo'. I think that could help you as it's the raw information you need. – LinuxSecurityFreak Nov 26 '19 at 18:29
  • @RafaelMuynarsk, Thanks for the tip :-) I will look into it. – sudodus Nov 26 '19 at 19:36
  • @LinuxSecurityFreak, Please write an answer. I would like to upvote and accept an answer by you :-) – sudodus Nov 27 '19 at 13:53
2

Shellscript

Kudos to @LinuxSecurityFreak for the advice to use the amount of 'Dirty' data reported in /proc/meminfo.

I made the following shellscript flusher. It shows progress and estimated time to flush the buffers. It can be used for example after cloning from an iso file to a USB drive or memory card in order to create a live drive with a linux operating system.

#!/bin/bash

timeorig=$(date '+%s')
deltat=5  # checking time interval

dirtorig=$(grep -e 'Dirty:' /proc/meminfo | tr -s ' ' '\t' | cut -f2)
dirt0=$dirtorig
echo -n "dirty = $dirt0 kB - before sync"

sync & spid=$!

while ps -A|grep "$spid" > /dev/null
do
 sleep "$deltat"
 dirty=$(grep -e 'Dirty:' /proc/meminfo | tr -s ' ' '\t' | cut -f2)
 deltad=$((dirt0-dirty))
 if [ $deltad -gt 0 ]
 then
  eta="$((dirty*deltat/deltad)) s"
  rate="$(((deltad+500)/deltat/1000)) MB/s"
 else
  eta="n.a."
  rate="n.a."
 fi
 echo -en "\0033[2K\0033[1G"
 echo -n "dirty = $dirty kB -- syncing -- rate = $rate -- eta = $eta"
 dirt0="$dirty"
done
echo -e "\0033[2K\0033[1GDone syncing :-)"

timefinal=$(date '+%s')
timeused=$((timefinal-timeorig))
if [ $timeused -gt 0 ]
then
 rate="$(((10*dirtorig+5)/timeused/10))"
 if [ $rate -ge 10000 ]
 then
  rate="$(((dirtorig+500)/timeused/1000)) MB/s"
 else
  rate="$rate kB/S"
 fi
else
 rate="n.a."
fi
echo "syncing time = $timeused s -- rate = $rate"

Demo examples

Cloning to a slow drive (USB 2)

$ sudo dd if=xubuntu-18.04.1-desktop-amd64.iso of=/dev/sde bs=1M ; ./flusher
[sudo] password for sudodus: 
1367+1 posts in
1367+1 posts ut
1434386432 bytes (1.4 GB, 1.3 GiB) copied, 0.408724 s, 3.5 GB/s

Output from flusher:

dirty = 840600 kB -- syncing -- rate = 5 MB/s -- eta = 156 s
...
Done syncing :-)
syncing time = 302 s -- rate = 4639 kB/S

Cloning to a fast drive (eSATA)

$ sudo dd if=xubuntu-18.04.1-desktop-amd64.iso of=/dev/sda bs=1M ; ./flusher
1367+1 posts in
1367+1 posts ut
1434386432 bytes (1.4 GB, 1.3 GiB) copied, 0.404442 s, 3.5 GB/s

Output from flusher:

dirty = 727508 kB -- syncing -- rate = 59 MB/s -- eta = 12 s
...
Done syncing :-)
syncing time = 25 s -- rate = 56 MB/s
1
  • 1
    Sorry for late reply. I managed to log in just now, was away the whole day, perfect answer. – LinuxSecurityFreak Nov 27 '19 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.