I have to transfer a 400Gb database consisting of a single file over the Internet from a server where I have full control to an other computer at the opposite border of the ocean (but which uses a slow connection). The transfer should take a full week and in order to reduce all protocol overhead (even using ftp would had 10Min overhead), I’m using a raw tcp connection.

I already transferred 10% of the file, but I learned there will be a scheduled outage in some hours.

How can I ask netcat or socat or curl to serve the FILE:database.raw at the offset it was interrupted? or

  • 1
    And this is why rsync is a disconcertingly good idea – roaima Nov 26 '19 at 13:22
  • @roaima even for a single file without having to care about file permissions? Isn’t rsync supposed to be a layer over tcp or udp? Also, no idea if I would be able to resume what’s already transfered that way. – user2284570 Nov 26 '19 at 13:24
  • Its big win is that it's (trivially) restartable – roaima Nov 26 '19 at 13:45
  • 1
    @roaima I don’t have physical access to the remote server. 400Gb is before compression but the data isn’t compression friendly so this doesn’t worth the time lost at compressing the file. – user2284570 Nov 26 '19 at 13:59
  • 1
    What commands did you use to transfer the first 10%? In order to resume the task, it would be nice to know how it was started. – Eduardo Trápani Nov 26 '19 at 15:04

If your command, as stated in the comments is:

socat -u FILE:somedatabase.raw TCP-LISTEN:4443,keepalive,tos=36

you can, on the sending side, do a seek and start serving from there:

socat -u gopen:somedatabase.raw,seek=1024 TCP-LISTEN:4443,keepalive,tos=36

on the receiving side you also need to seek:

socat tcp:example.com:4443 gopen:somedatabase.raw,seek=1024

Check the manpage for socat, there are other options you might be interested in.

  • Thank you. How do I get the seek value ? – user2284570 Nov 26 '19 at 16:02
  • Check the how many bytes you received so far and use that value. An ls -l should do it. – Eduardo Trápani Nov 26 '19 at 16:07
  • I mean shouldn’t I add 1 or remove 1 to the value or ls -l in order to get the seek value ? – user2284570 Nov 26 '19 at 16:20
  • 1
    I checked it with the same value I got with ls. Be aware that you cannot run that multiple times because it will append. In that case, if you are not sure, you could also check the ftruncate flag in socat. seek-end can be also useful. You've taken the one of the hardest roads, so be prepared for some bumps. If you feel like testing your commands, using a small text file might help you see how all those parameters interact, though you should run something like md5sum to verify. – Eduardo Trápani Nov 26 '19 at 16:38
  • Last question, am I correct to assume that a transfer using IP4-LISTEN of socat will produce an incorrect result in even if it’s the only communication which is happening with the remote server ? – user2284570 Nov 26 '19 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.