2

I'm having trouble with a large text file (30GB)

I would like to create smaller files from it (5GB lets say)

But sadly I don't have no more storage (only ~10G is available).

This line:

split -b 5g "file.txt" "file.txt."

Leaves the original file in place, meaning I will end up with 60 GB in total. Which is more than I actually have free left in my pc.

How could I run a code that does the same, while not keeping the original file?

2
  • Note that split -b works on bytes: it will split a few lines across two files. From the man page, split -c keeps any whole lines together. Jan 12, 2021 at 10:45
  • If you have a back-up device, you must surely have enough free space on there, because you would need it for your next complete backup. If you don't have a back-up device, buy one. Today! It will be about 20 bucks. Jan 12, 2021 at 14:09

1 Answer 1

2

On a file system having only ~1.03G of free-disk-space;
splitting a file ~9.3G of size.

we are going to split this file into 10 small files each about 1G of size.

for i in {1..10}; do
    tail -c 1G bigfile >split_file_$i && truncate -s -1G bigfile;
done

this almost use 1G of free-disk space from ~1.03G available and only ~30M remaining each time! (I'm on test environment); of course you need adjust split size according to your available free-disk-space and don't go to use all of that available space!, use for example up-to 80% of it.


to rejoin back into one file:

for file in split_file_{10..1}; do
    cat "$file" >>bigfile_joined && rm "$file"; 
done
1
  • 1
    A minor optimisation: the first section of the original file (i.e. the last part to be processed) does not need to be copied, as it has already been truncated nine times: renaming it would suffice. With 10 sections, that is only a 10% improvement (less if the size division is unequal). With two or three sections, it may be a significant improvement. It also avoids having to remove the (now zero-sized) bigfile. Jan 14, 2021 at 11:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .