So I have a 100GB text files And I want to split it into 10000 files. I used to do such tasks with something like:
split -l <number of lines> -d --additional-suffix=.txt bigfile small_files_prefix
But I tried to do that with this one and I monitored my system and realized that it wasn't using much memory or CPU so I realized that it's just reading the file from beginning to end with one thread.
Is there any low level(or very high performance) tool that can do such a task with multiple threads.
I would even prefer to copy the file if necessary and take advantage of my multiple cores if possibly faster(I don't think so!).