I'm having trouble with a large text file (30GB)

I would like to create smaller files from it (5GB lets say)

But sadly I don't have no more storage (only ~10G is available).

This line:

split -b 5g "file.txt" "file.txt."

Leaves the original file in place, meaning I will end up with 60 GB in total. Which is more than I actually have free left in my pc.

How could I run a code that does the same, while not keeping the original file?

  • Note that split -b works on bytes: it will split a few lines across two files. From the man page, split -c keeps any whole lines together. Jan 12 '21 at 10:45
  • If you have a back-up device, you must surely have enough free space on there, because you would need it for your next complete backup. If you don't have a back-up device, buy one. Today! It will be about 20 bucks. Jan 12 '21 at 14:09

On a file system having only ~1.03G of free-disk-space;
splitting a file ~9.3G of size.

we are going to split this file into 10 small files each about 1G of size.

for i in {1..10}; do
    tail -c 1G bigfile >split_file_$i && truncate -s -1G bigfile;

this almost use 1G of free-disk space from ~1.03G available and only ~30M remaining each time! (I'm on test environment); of course you need adjust split size according to your available free-disk-space and don't go to use all of that available space!, use for example up-to 80% of it.

to rejoin back into one file:

for file in split_file_{10..1}; do
    cat "$file" >>bigfile_joined && rm "$file"; 
  • A minor optimisation: the first section of the original file (i.e. the last part to be processed) does not need to be copied, as it has already been truncated nine times: renaming it would suffice. With 10 sections, that is only a 10% improvement (less if the size division is unequal). With two or three sections, it may be a significant improvement. It also avoids having to remove the (now zero-sized) bigfile. Jan 14 '21 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.