Can I watch the progress of a `sync` operation?

Question

I've copied a large file to a USB disk mounted on a Linux system with async. This returns to a command prompt relatively quickly, but when I type sync, of course, it all has to go to disk, and that takes a long time.

I understand that it's going to be slow, but is there somewhere where I can watch a counter go down to zero? Watching buffers in top doesn't help.

Answer 1

Looking at /proc/meminfo will show the Dirty number shrinking over time as all the data spools out; some of it may spill into Writeback as well. That will be a summary against all devices, but in the cases where one device on the system is much slower than the rest you'll usually end up where everything in that queue is related to it. You'll probably find the Dirty number large when you start and the sync finishes about the same time it approaches 0. Try this to get an interactive display:

watch -d grep -e Dirty: -e Writeback: /proc/meminfo

With regular disks I can normally ignore Writeback, but I'm not sure if it's involved more often in the USB transfer path. If it just bounces up and down without a clear trend to it, you can probably just look at the Dirty number.

Answer 2

You can look at the /sys/block/<device>/stat file for the appropriate device while you're syncing. The 9^th column will indicate the number of in-flight requests on the device, which should go down to zero when the sync is done.
Don't know of a way to translate that to a number of bytes, but it should give you a rough idea of how much "stuff" is still pending.

See the stat.txt file in the kernel documentation for a bit more information. (There's also an inflight file in that directory on my system which looks like it could contain read and write in-flight requests, but I can't find docs for that.)

Answer 3

By using Greg's answer, you can simply have sync run in background while displaying the state of the Dirty block in memory.

To achieve this, simply run this command:

sync & watch -n 1 grep -e Dirty: /proc/meminfo

This will call sync in the background while executing watch in the front. When the sync command will have finished (around when the size of the Dirty block has reached 0), you will have an output that looks like this :

1]  + 27260 done        sync

This means that the command has finished and you can kill the watch command with Ctrl+C.

Answer 4

Based on the other answers I made this over-bloated one-liner that tracks summary progress as well as device-specific progress:

# Create the command
watchSync() {
  watch -n1 'grep -E "(Dirty|Write)" /proc/meminfo; echo; ls /sys/block/ | while read device; do awk "{ print \"$device: \"  \$9 }" "/sys/block/$device/stat"; done'
}

# Run the command
watchSync

The output looks like this:

Dirty:            848956 kB
Writeback:        125948 kB
WritebackTmp:          0 kB

dm-0: 0
dm-1: 0
loop0: 0
loop1: 0
loop2: 0
loop3: 0
loop4: 0
loop5: 0
loop6: 0
loop7: 0
nvme0n1: 0
sda: 0
sdb: 0
sdc: 124

The above tells me that 1) I have a few 100,000 kBytes that need writing, and 2) the device being written to is sdc. Sync is complete when, in this case, sdc and Writeback hit zero (this will happen at the same time).

Answer 5

You can watch current block device I/O traffic with nmon, as in:

NMON=ld nmon -s1

(this preselects display of load graph and disk graph with 1 second refresh time.. this can also be set by starting nmon without parameters, than pressing l, d, - to toggle graphs and reduce refresh time from default 2s .. + of course would increase refresh delay.)

Answer 6

I wrote a script for this based on the answers here:

watchSync

#!/bin/sh

# Output the `Dirty` section of `/proc/meminfo`
report() {
  local dirty="$(grep -w Dirty: /proc/meminfo | cut -d ':' -f 2 | tr -d ' ')"
  echo -n -e "\e[2K\rSyncing ${dirty}... "
}

report
# Start syncing
sync &
SYNC_PID=$!
# Give a short sleep in case there's not much
sleep 0.125
# While sync is running, report
while ps -p $SYNC_PID > /dev/null 2>&1; do
  report
  sleep 1
done

echo "Done!"

Answer 7

repr () {
    if [[ $# -gt 0 ]]; then
        local result="$(printf -- '%q ' "$@")"
        printf -- '%s' "${result::-1}"
    fi
    return $?
}

sync () {
    local args
    if [[ $- = *i* ]]; then
        args="$(repr "$@")"
        bash -sc "$(shopt -p); $(declare -f); ${FUNCNAME[0]} $args"
        return $?
    fi
    local initial_value
    local percentage
    local backspace=''
    local i
    command sync "$@" &
    initial_value="$(grep -e 'Dirty:' /proc/meminfo | awk '{print $2}')"
    if [[ $initial_value = 0 ]]; then initial_value=1; fi
    sleep 0.1
    while ps -p $! > /dev/null; do
        percentage="$(((($initial_value-$(grep -e 'Dirty:' /proc/meminfo | awk '{print $2}'))*100)/$initial_value))"
        percentage="$(( $percentage < 0 ? 0 : $percentage ))%"
        printf -- '%b' "$backspace$percentage"
        backspace=''
        for (( i=1; i<=${#percentage}; i++ )); do backspace+='\b \b'; done
        sleep 1
    done
    printf -- '%b' "$backspace"
    return $(wait $!)
}

Answer 8

If anyone is looking for any tools that automate rsync and sync monitoring afterwards. I've made one, based on others great answers, check it out:

https://github.com/satk0/usbdrivetools

Using the tool you can simply run:

usbcp /path/to/src /path/to/dest

To have a source file copied to USB Drive mounted destination.