How to read only the last line of a live input stream in a while read loop

Question

I have a very simple bash script that synchronizes data to a USB key as soon as new files appear. It is poorly written but it works, however I want to minimize CPU time so I used sleep but since I use inotifywait as a trigger for synchronization, the script adds one second of sleep for every new file added (and there are many small files). My solution would be that read only reads the latest line of inotifywait output, which I wanted to do by piping it into tail -1, but it doesn't seem to work.

#!/bin/bash

watched_path=".wine/drive_c/Data"
rm -r "$HOME/$watched_path/"*
inotifywait -r -m "$HOME/$watched_path" --format "%e %w%f" -e create -e moved_to |
  while read action full_path; do
    sleep 1
    file=${full_path##*$watched_path}
    echo "'$file' appeared in directory '$watched_path' via '$action'"
    USB="USBKEY"

    # Wait for USB to be connected before syncing.
    until [ -e /media/$USER/$USB ]; do
      echo "Connect USB for sync."
      sleep 5
    done
    rsync -r "$HOME/$watched_path" /media/$USER/$USB
    echo "Sync complete."
  done

I sync the whole folder so it's okay if I miss a line of inotifywait, but I need it to be synced after the latest file (and thus latest line) at the moment of reading. This would mean that from the moment I stop sending files, the while loop will be executed at most one more time before hanging again.

How can I read only the most recent line of the inotifywait output? If you have other suggestions avoiding this issue it's okay for me but I'm interested in the answer nonetheless.

Answer 1

I think you're asking for a more complex scenario. You haven't described how to identify the "last" file to arrive, so this one simply times out after 5 seconds of file inactivity since at least one file has been transferred.

1. Set flag = waiting
2. In parallel: whenever a file is notified, set flag = not_done

3. Wait until flag == not_done

4. Start loop

   • Wait for USB to be connected and start the synchronisation process (rsync)
   • If any new files were transferred then set flag = not_done
   • Otherwise
     • If flag == probably_done then set flag = definitely_done otherwise set flag = probably_done
   • If we took less than five seconds then wait for the remaining time
5. Loop until flag == definitely_done

Since we need IPC between the main process and the parallel inotifywait process I'd consider implementing the flag as a file, possibly even a set of file names.

As far as I can tell it's not easy to identify whether or not rsync has transferred new files. You might need to compare target directory contents before and after.