3

I have to transfer a 400Gb database consisting of a single file over the Internet from a server where I have full control to an other computer at the opposite border of the ocean (but which uses a slow connection). The transfer should take a full week and in order to reduce all protocol overhead (even using ftp would had 10Min overhead), I’m using a raw tcp connection.

I already transferred 10% of the file, but I learned there will be a scheduled outage in some hours.

How can I ask netcat or socat or curl to serve the FILE:database.raw at the offset it was interrupted? or

Improve this question
13
  • 1
    And this is why rsync is a disconcertingly good idea
    – Chris Davies
    Commented Nov 26, 2019 at 13:22
  • @roaima even for a single file without having to care about file permissions? Isn’t rsync supposed to be a layer over tcp or udp? Also, no idea if I would be able to resume what’s already transfered that way.
    – user2284570
    Commented Nov 26, 2019 at 13:24
  • 1
    The other question, which isn't really technical, is whether it would be quicker/easier to post a CD
    – Chris Davies
    Commented Nov 26, 2019 at 13:56
  • 1
    @roaima I don’t have physical access to the remote server. 400Gb is before compression but the data isn’t compression friendly so this doesn’t worth the time lost at compressing the file.
    – user2284570
    Commented Nov 26, 2019 at 13:59
  • 1
    What commands did you use to transfer the first 10%? In order to resume the task, it would be nice to know how it was started.
    – Eduardo Trápani
    Commented Nov 26, 2019 at 15:04

1 Answer 1

Reset to default
2

If your command, as stated in the comments is:

socat -u FILE:somedatabase.raw TCP-LISTEN:4443,keepalive,tos=36

you can, on the sending side, do a seek and start serving from there:

socat -u gopen:somedatabase.raw,seek=1024 TCP-LISTEN:4443,keepalive,tos=36

on the receiving side you also need to seek:

socat tcp:example.com:4443 gopen:somedatabase.raw,seek=1024

Check the manpage for socat, there are other options you might be interested in.

Improve this answer
7
  • Thank you. How do I get the seek value ?
    – user2284570
    Commented Nov 26, 2019 at 16:02
  • Check the how many bytes you received so far and use that value. An ls -l should do it.
    – Eduardo Trápani
    Commented Nov 26, 2019 at 16:07
  • 1
    I mean shouldn’t I add 1 or remove 1 to the value or ls -l in order to get the seek value ?
    – user2284570
    Commented Nov 26, 2019 at 16:20
  • 2
    I checked it with the same value I got with ls. Be aware that you cannot run that multiple times because it will append. In that case, if you are not sure, you could also check the ftruncate flag in socat. seek-end can be also useful. You've taken the one of the hardest roads, so be prepared for some bumps. If you feel like testing your commands, using a small text file might help you see how all those parameters interact, though you should run something like md5sum to verify.
    – Eduardo Trápani
    Commented Nov 26, 2019 at 16:38
  • Last question, am I correct to assume that a transfer using IP4-LISTEN of socat will produce an incorrect result in even if it’s the only communication which is happening with the remote server ?
    – user2284570
    Commented Nov 26, 2019 at 16:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .