How to meter FTP upload progress in telnet

Question

I'm working on a script using telnet to connect to an ftp server and dropping a big file on it.

Because of the environment of the project, I need to use ftp and cannot rely on other techniques like scp, rsync or whatever.

How can I know the progress of my upload?

I can switch server side between pureftp and proftpd.

I tried to analyse how far I am in writing to transfer channel, but this seems wrong as it shows finished long before the file is on the server. So I'd like to have some feedback server side. I was hoping to find some magical STOR command giving feedback on how much data was actually written.

I've found the STAT command on proftpd but it looks laggy and needs to be requested.

Is there an easy way to achieve this progress?

How does the traditional HASH feedback on client-side ftp works?

Answer 1

The behaviour you are observing happens because even though the client has written data to the socket for the FTP data connection and the client's kernel has replied that this happened successfully (by returning successfully from the write() or send() system call), the data is still in all sort of buffers and not already committed to the destination file. It's in network buffers in the source OS, it's in buffers in ethernet cards, it's in flight on the wire, it's in buffers at the destination, it's in the FTP server process' buffers, it's been written to a file on the server but not flushed to disk, etc...

It doesn't only happen with FTP. You can easily notice this happening with more modern file transfer tools like scp too. scp a file halfway aqround the planet (for a high-latency network) or to a system with slow storage, and you will see that the transfer progress is shown as faster than it really is, then it seems to hang at 100% complete for a while before the command actually completes.

If you look at the source code for netkit-ftp (the default/standard FTP client on Debian), its # mark processing does not do anything special. It writes the hash marks according to how much data it has written to the data connection socket. Thus, it will exhibit the same behaviour you are observing.

What normal FTP clients do, and what you should do too, is wait for the reply to the STOR command on the control connection. This reply will come some time after you have already flushed and closed the data connection socket. When you get that reply, you know that the FTP server has the data. It might not be fully committed to disk over there, but that's as much of a confirmation as you're going to get.

Answer 2

If your client/server doesn't show progress - you can check progress of upload on client side, based on progress of file reading.

1. you have size of uploaded file
2. you know pid process which upload file (for example 1234).
3. you can look into /proc/1234/fd - which fd belongs to readed file (for example 5).
4. then - you can look into /proc/1234/fdinfo/5 - you see here something like

pos:    12313
flags:  0100002

It mean, that position in this file is 12313 - so - if you divide position (12313) by filesize - you have overall progress.