I am trying to fetch a log file which is repeatedly updated. I want that updated file, but don't want to download the entire file again and again from the remote system via SCP.
Any idea to achieve this in minimal network usage?
With SCP alone, there isn't really any solution. The protocol only allows for full file copies, not partial transfers.
If you're willing to look beyond just SCP, there are a couple of options:
Use a command like the following to generate a file containing just the new lines of the log file and transfer that:
ssh user@host tail -n +`wc -l /local/file.log` /remote/file.log >> /local/file.log
That will count the lines in the local file, and tell the remote system to print all lines after that line number over the SSH connection, which will then be appended to the local file by shell redirection. This method may be a bit finicky in some shells, and is slightly racy (a write arriving at the file while it's being copied may end up only partially transferred, but won't get completely transferred by the next invocation).
rsync with the
--checksum option. This enables incremental transfers of data, and because of the rolling-hash checksumming it uses should transfer an absolute minimum of data. Unlike the first method, this isn't racy at all, but it also requires rsync being present on both ends.
Set up a (secured) web server that supports byte-range requests. You can then use an HTTP HEAD request to get the new file size (optionally with an If-Modified-Since header to turn it into a no-op if the file hasn't changed), and request the range extending from where your copy ends to the total length reported by the HEAD request. This has the most setup involved, and can transfer partial lines, but will guarantee you can't lose any data.
Figure out a way to set up remote logging for the application generating the log file, and set up your local system as a log host. This only works of course if your local system is on all the time, but it also guarantees that you have the most up-to-date version of the log possible without you needing to do anything by hand to update it.