How can I estimate the whole time of a write process including sync?

Question

Progress and estimated time to write without and with sync

I have found no tool (or straightforward method) that will include flushing the buffers when showing the progress and estimating the estimated time for the whole write process, ETA (Estimated Time of Arrival).

• pv can show the time for the progress as seen by the operating system, but if the target drive is slow and there is a lot of RAM, it shows only the time until the data are written to a buffer. This time can be a small fraction of the the real time until the buffers are flushed.

• dd writes a final report about amount of data used time and transfer rate. It can also be made to write 'progress' reports. It used to give a much better estimate than pv, but nowadays the USB drives and memory cards are still very slow, while the other processes are fast and the available memory for buffers big. So dd will also finish long before the buffers are flushed.

• I can 'time' the write process including sync with the time command

  time ( write command; sync )
  

  and it will give me the real time used which is useful, but only after it has finished. It does not show the progress and does not estimate the total remaining time.

• I can run iotop to show read and write processes and how fast things are read and written, but it does not estimate the remaining time.

How to show progress and estimated time for the whole write process?

How can I show progress and estimated time for the whole write process, ETA (Estimated Time of Arrival), including flushing the buffers with sync?

Link to related question

• Understanding sync command operations in linux

Answer 1

Shellscript

Kudos to @LinuxSecurityFreak for the advice to use the amount of 'Dirty' data reported in /proc/meminfo.

I made the following shellscript flusher. It shows progress and estimated time to flush the buffers. It can be used for example after cloning from an iso file to a USB drive or memory card in order to create a live drive with a linux operating system.

#!/bin/bash

timeorig=$(date '+%s')
deltat=5  # checking time interval

dirtorig=$(grep -e 'Dirty:' /proc/meminfo | tr -s ' ' '\t' | cut -f2)
dirt0=$dirtorig
echo -n "dirty = $dirt0 kB - before sync"

sync & spid=$!

while ps -A|grep "$spid" > /dev/null
do
 sleep "$deltat"
 dirty=$(grep -e 'Dirty:' /proc/meminfo | tr -s ' ' '\t' | cut -f2)
 deltad=$((dirt0-dirty))
 if [ $deltad -gt 0 ]
 then
  eta="$((dirty*deltat/deltad)) s"
  rate="$(((deltad+500)/deltat/1000)) MB/s"
 else
  eta="n.a."
  rate="n.a."
 fi
 echo -en "\0033[2K\0033[1G"
 echo -n "dirty = $dirty kB -- syncing -- rate = $rate -- eta = $eta"
 dirt0="$dirty"
done
echo -e "\0033[2K\0033[1GDone syncing :-)"

timefinal=$(date '+%s')
timeused=$((timefinal-timeorig))
if [ $timeused -gt 0 ]
then
 rate="$(((10*dirtorig+5)/timeused/10))"
 if [ $rate -ge 10000 ]
 then
  rate="$(((dirtorig+500)/timeused/1000)) MB/s"
 else
  rate="$rate kB/S"
 fi
else
 rate="n.a."
fi
echo "syncing time = $timeused s -- rate = $rate"

Demo examples

Cloning to a slow drive (USB 2)

$ sudo dd if=xubuntu-18.04.1-desktop-amd64.iso of=/dev/sde bs=1M ; ./flusher
[sudo] password for sudodus: 
1367+1 posts in
1367+1 posts ut
1434386432 bytes (1.4 GB, 1.3 GiB) copied, 0.408724 s, 3.5 GB/s

Output from flusher:

dirty = 840600 kB -- syncing -- rate = 5 MB/s -- eta = 156 s
...
Done syncing :-)
syncing time = 302 s -- rate = 4639 kB/S

Cloning to a fast drive (eSATA)

$ sudo dd if=xubuntu-18.04.1-desktop-amd64.iso of=/dev/sda bs=1M ; ./flusher
1367+1 posts in
1367+1 posts ut
1434386432 bytes (1.4 GB, 1.3 GiB) copied, 0.404442 s, 3.5 GB/s

Output from flusher:

dirty = 727508 kB -- syncing -- rate = 59 MB/s -- eta = 12 s
...
Done syncing :-)
syncing time = 25 s -- rate = 56 MB/s

Edit:

There is an updated version of flusher, called watch-flush, in the current version in mkusb. It can be used separately from an own window via the following alias,

alias wf='xterm -title "watch-flush" -fa default -fs 13 -fg yellow -bg "#504030" -geometry 70x7 -e bash -c "watch-flush;read -n1" 2> /dev/null'