I am trying to fetch a log file which is repeatedly updated. I want that updated file, but don't want to download the entire file again and again from the remote system via SCP.

Any idea to achieve this in minimal network usage?

  • 2
    SCP protocol does not allow partial files downloads. Do you really need to use SCP? What about SFTP? Or a shell access? – Martin Prikryl Jan 29 '18 at 17:10
  • rsync only moves changed bytes... or HTTP has partial content obtaination – thrig Jan 29 '18 at 17:11
  • rsync? That would only transfer the changed bits. – Kusalananda Jan 29 '18 at 17:12
  • The hard part is how to know which part of the file to be copied file has already been transferred. You will either have to save some kind of cursor into the file to be able to seek to the start of the newly added parts of the file, or you will have to implement a continuous process which polls the file and sends the new bits when they arrive. Perhaps something along the lines of ssh remotehost "tail -f logfile" > localcopy? – Johan Myréen Jan 29 '18 at 17:48

With SCP alone, there isn't really any solution. The protocol only allows for full file copies, not partial transfers.

If you're willing to look beyond just SCP, there are a couple of options:

  1. Use a command like the following to generate a file containing just the new lines of the log file and transfer that:

    ssh user@host tail -n +`wc -l /local/file.log` /remote/file.log >> /local/file.log

    That will count the lines in the local file, and tell the remote system to print all lines after that line number over the SSH connection, which will then be appended to the local file by shell redirection. This method may be a bit finicky in some shells, and is slightly racy (a write arriving at the file while it's being copied may end up only partially transferred, but won't get completely transferred by the next invocation).

  2. Use rsync with the --checksum option. This enables incremental transfers of data, and because of the rolling-hash checksumming it uses should transfer an absolute minimum of data. Unlike the first method, this isn't racy at all, but it also requires rsync being present on both ends.

  3. Set up a (secured) web server that supports byte-range requests. You can then use an HTTP HEAD request to get the new file size (optionally with an If-Modified-Since header to turn it into a no-op if the file hasn't changed), and request the range extending from where your copy ends to the total length reported by the HEAD request. This has the most setup involved, and can transfer partial lines, but will guarantee you can't lose any data.

  4. Figure out a way to set up remote logging for the application generating the log file, and set up your local system as a log host. This only works of course if your local system is on all the time, but it also guarantees that you have the most up-to-date version of the log possible without you needing to do anything by hand to update it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.