9

Some background you can happily skip

Twenty years ago or so, when navigating the web costed a lot, when I was a Windows-only user, and when CDs/DVDs were a large storage means, and when sharing video files with a friend or relative would sometimes require to split the file over multiple CDs/DVDs, copying them on the other computer's person and then rejoining the pieces, I used to use HJSplit. Worked like a charm.

The motivation

Fast-forward 20 years, I recently found myself in need for such a utility on Linux, due to the slow/unreliable connection not allowing me to easily scp stuff across physically very distant Linux systems. The solution that came to mind was to split the file and transfer the pieces, then rejoin them.

That's how I found HJSplit was Windows-only and that lxsplit existed and worked like a charm as well, so all is good.

My question

But lxsplit is abandoned since 2008, so maybe some other (better?) solution has come up in these 15 years.

What is the state of the art in this field, i.e. splitting and rejoining big binary files, on Linux?¹

Additional motivation

I also thought that conceptually speaking, splitting a file and rejoining it is a very simple task, so I wondered whether I could write my own program for doing so. I tried, and got something working in a few hours, but it's at least ~5 times slower than lxsplit. Before diving into profiling and benchmarking, I wanted to know whether there's other similar programs that have even better performance than lxsplit.

(¹) I'm not interested in alternative workflows for accomplishing the original task of transfering a big file between two systems. Yes, today you'd probably upload it to Dropbox/Onedrive/GoogleDrive/whatever from one system and download it from the other.

Improve this question
19
  • 7
    Have you check split command?
    – Romeo Ninov
    Jul 16 at 6:54
  • 15
    Have you considered rsync, a restartable copy command?
    – Chris Davies
    Jul 16 at 7:01
  • 2
    @RomeoNinov, I did not know of it, thanks.
    – Enlico
    Jul 16 at 7:17
  • 7
    I read your note and I decided it would still be remiss of me not to mention rsync. It's simpler and more efficient to use than either of your proposed transfer mechanisms
    – Chris Davies
    Jul 16 at 7:23
  • 5
    For something so simple, it shouldn't matter whether the software was last updated this year, 10 years ago, or 30 years ago.
    – hobbs
    Jul 17 at 2:59

3 Answers 3

Reset to default
25

The split command has been part of Unix since the ancient days, and while it was originally a text processing command that split in lines, modern implementations also work with binary files. split -b is in POSIX and BusyBox.

To split, choose a size for the pieces (the last piece will be whatever's left) and a prefix to the file names of the pieces. (You don't get to choose the numbering style or an extension for the file name.)

split -b 1440k myfile.bin part-

To concatenate the pieces, just use cat. Use a shell wildcard for the pieces and they'll be in the correct order.

cat part-?? >myfile.bin

Note that you're guaranteed to get the parts in the correct order, but there's no check for missing parts. Also there's no protection against mixing parts from different files. Use a command such as cksum, md5sum, md5, or whatever is available on both systems to calculate a checksum of the original file and the reassembled file.

Improve this answer
2
  • (As regards the concactenation,) I might have to double check and test for different file sizes, but earlier I gave a few tries to lxsplit -j part1 vs cat part* and the former turned out to be faster.
    – Enlico
    Jul 16 at 11:40
  • 3
    @DmitryGrigoryev With split, the order from shell wildcard expansion is guaranteed to be correct, because it uses a fixed-length sequence of letters, not a variable-length sequence of digits.
    – Gilles 'SO- stop being evil'
    Jul 17 at 14:58
1

The solution that came to mind was to split the file and transfer the pieces, then rejoin them.

Then this sounds like the XY problem. Just use rsync:

rsync -avzh verybigfileordirectory [email protected]:/home/user/destination/

You can stop this and start it as required. It will work out what data is missing at the remote end (small negotiation overhead) and copy only that. It does require rsync at both ends.

Improve this answer
1
  • There's no XY problem. Refer to the last paragraph in my question + the footnote. But thanks for pointing out what was already pointed out in the comments to my question.
    – Enlico
    Aug 5 at 7:26
0

I'd suggest zip, which is specifically oriented towards binary files.

Although some users might be wary of it due to a certain redolence of DOS :-)

Improve this answer
18
  • 2
    As I said, my interest is specifically in splitting a file in chunks.
    – Enlico
    Jul 16 at 18:25
  • 2
    Yes, which zip does very nicely.
    – Mark Morgan Lloyd
    Jul 16 at 20:10
  • 19
    Then, please, add code to support the answer. Zip is notoriously known to compress files. If it does more, then clarify.
    – Enlico
    Jul 16 at 20:24
  • 4
    @MarkMorganLloyd, what do you mean that split wasn't "written to support" binary files? It has a byte-by-byte mode; what else is needed to support binary files? It's not like UNIX is Windows with horrors like automated newline conversion that make binary and text different from each other; particularly back in the 70s when split was designed and Unicode wasn't a thing yet, a byte is a byte is a byte. (Yes, you had character sets/encodings, but everyone worked in 8 bits so it didn't matter to a tool like split).
    – Charles Duffy
    Jul 17 at 13:42
  • 3
    @MarkMorganLloyd, right, but traditional UNIX systems don't have a meaningful text/binary distinction. split supports splitting by line, which is of course textual in nature, but because the OS's file I/O doesn't care, that doesn't change anything at all about how its byte-count-based methods work.
    – Charles Duffy
    Jul 17 at 15:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .