Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

How do I break a large, +4GB file into smaller files of about 500MB each.

And how do I re-assemble them again to get the original file?

share|improve this question

3 Answers 3

up vote 22 down vote accepted

You can use split and cat.

For example something like

$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.

(where the input filename is foo and the last argument is the output prefix)

The same command with short options:

$ split -b 100k -d -a 3 foo foo.

The split commands generate pieces named: foo.000, foo.001 ...

For re-assembling the generated pieces again you can use e.g.:

$ cat foo.* > foo_2

(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)

You can compare the result via:

$ cmp foo foo_2
$ echo $?

(which should output 0)

Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:

$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3
share|improve this answer
2  
Try this command: man split cat md5sum –  Kevin M Sep 4 '10 at 19:13
4  
When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc. –  Gilles Oct 17 '10 at 11:16
1  
And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc. –  Zorawar Nov 29 '12 at 18:05

You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.

share|improve this answer
1  
i tagged it 'command-line', but thanks for the answer :) –  Stefan Sep 5 '10 at 19:25

lxsplit does splitting and joining allowing you to specify chunk size. It can handle files larger than 4Gb.

lxsplit --help

LXSplit v0.2.4 by Richard Stellingwerff, O. Sezer.
Home page: http://lxsplit.sourceforge.net/

Usage: lxsplit [OPTION] [FILE] [SPLITSIZE]

Available options:
 -j : join the files beginning with the given name
 -s : split the given file.  requires a valid size
Splitsize examples: 15M, 100m, 5000k, 30000000b

Examples:
    lxsplit -s hugefile.bin 15M
    lxsplit -j hugefile.bin.001
share|improve this answer
2  
"It can handle files larger than 4Gb." - suggests another solution can't, but I don't think there's such a limit on them. –  gertvdijk Dec 8 '12 at 0:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.