2

So I have a 100GB text files And I want to split it into 10000 files. I used to do such tasks with something like:

split -l <number of lines> -d --additional-suffix=.txt bigfile small_files_prefix

But I tried to do that with this one and I monitored my system and realized that it wasn't using much memory or CPU so I realized that it's just reading the file from beginning to end with one thread.

Is there any low level(or very high performance) tool that can do such a task with multiple threads.

I would even prefer to copy the file if necessary and take advantage of my multiple cores if possibly faster(I don't think so!).

2

Even with SSDs the bottleneck of splitting files is I/O. Having several processes / threads for that will not gain performance and often be much slower.

In addition if you want to split on newlines only then it is not clear in advance from where to where each thread has to copy. You would probably have to write a special tool for that.

The situation might be different if another action is needed like e.g. splitting and compressing. In that case the use of several cores might help but then I/O is not the bottleneck (depending on drive and CPU speed).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.