0

I have a very simple bash script that synchronizes data to a USB key as soon as new files appear. It is poorly written but it works, however I want to minimize CPU time so I used sleep but since I use inotifywait as a trigger for synchronization, the script adds one second of sleep for every new file added (and there are many small files). My solution would be that read only reads the latest line of inotifywait output, which I wanted to do by piping it into tail -1, but it doesn't seem to work.

#!/bin/bash

watched_path=".wine/drive_c/Data"
rm -r "$HOME/$watched_path/"*
inotifywait -r -m "$HOME/$watched_path" --format "%e %w%f" -e create -e moved_to |
  while read action full_path; do
    sleep 1
    file=${full_path##*$watched_path}
    echo "'$file' appeared in directory '$watched_path' via '$action'"
    USB="USBKEY"

    # Wait for USB to be connected before syncing.
    until [ -e /media/$USER/$USB ]; do
      echo "Connect USB for sync."
      sleep 5
    done
    rsync -r "$HOME/$watched_path" /media/$USER/$USB
    echo "Sync complete."
  done

I sync the whole folder so it's okay if I miss a line of inotifywait, but I need it to be synced after the latest file (and thus latest line) at the moment of reading. This would mean that from the moment I stop sending files, the while loop will be executed at most one more time before hanging again.

How can I read only the most recent line of the inotifywait output? If you have other suggestions avoiding this issue it's okay for me but I'm interested in the answer nonetheless.

  • 1
    How do you know it’s the last file? – Jeff Schaller Sep 24 '18 at 12:06
  • from man inotifywait: -m: Instead of exiting after receiving a single event, execute indefinitely. – Jeff Schaller Sep 24 '18 at 12:36
  • Well I'm assuming if I read the most recent line comming from inotifywait it's going to be the last file that was added. – Flabou Sep 24 '18 at 19:42
  • Why couldn’t another file show up from the inotifywait output? – Jeff Schaller Sep 24 '18 at 19:42
  • I'm sorry, I explained myself badly and chose my words badly, it's the latest file that I'm trying to get, but there can always be new files comming, (the script runs forever and a new batch of files can come at any time). I'll edit my question to make this more clear. – Flabou Sep 24 '18 at 19:47
0

I think you're asking for a more complex scenario. You haven't described how to identify the "last" file to arrive, so this one simply times out after 5 seconds of file inactivity since at least one file has been transferred.

  1. Set flag = waiting
  2. In parallel: whenever a file is notified, set flag = not_done
  3. Wait until flag == not_done

  4. Start loop

    • Wait for USB to be connected and start the synchronisation process (rsync)
    • If any new files were transferred then set flag = not_done
    • Otherwise
      • If flag == probably_done then set flag = definitely_done otherwise set flag = probably_done
    • If we took less than five seconds then wait for the remaining time
  5. Loop until flag == definitely_done

Since we need IPC between the main process and the parallel inotifywait process I'd consider implementing the flag as a file, possibly even a set of file names.

As far as I can tell it's not easy to identify whether or not rsync has transferred new files. You might need to compare target directory contents before and after.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.