14

This question already has an answer here:

Is there a multi-threaded cp command on Linux?

I know how to do this on Windows, but I don't know how this is approached in a Linux environment.

marked as duplicate by Anthon, jimmij, slm Nov 1 '14 at 0:58

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

27

As Celada mentioned, there would be no point to using multiple threads of execution since a copy operation doesn't really use the cpu. As ryekayo mentioned, you can run multiple instances of cp so that you end up with multiple concurrent IO streams, but even this is typically counter-productive. If you are copying files from one location to another on the same disk, trying to do more than one at a time will result in the disk wasting time seeking back and forth between each file, which will slow things down. The only time it is really beneficial to copy multiple files at once is if you are, for instance, copying several files from several different slow, removable disks onto your fast hard disk, or vice versa.

  • 2
    Disk seeks are only relevant for non-SSD disks. – Thorbjørn Ravn Andersen Oct 31 '14 at 19:34
  • 2
    @ThorbjørnRavnAndersen: but random seeks tend to interact badly with caches (in particular, read-ahead caches) for any kind of medium. – Matteo Italia Oct 31 '14 at 23:59
  • @ThorbjørnRavnAndersen, the severe penalty for seeks on HDD is almost none on SSD, yet the point remains that there is no benefit to trying to read or write to/from multiple parts of a single disk at the same time. – psusi Nov 1 '14 at 2:18
  • @MatteoItalia if the IO-channel is saturated there is nothing caches can improve. – Thorbjørn Ravn Andersen Nov 1 '14 at 9:49
  • @psusi The argument was that the disk wasted time seeking, not that the disk could not serve data any faster. – Thorbjørn Ravn Andersen Nov 1 '14 at 9:51
16

Well, I believe you could use gnu parallel to accomplish your task.

 seq 70 | parallel -j70 cp filename

You could see a detailed explanation on using gnu parallel from my other answer here.

I just tested the above command in my system and I could see that 70 copies of files are being made.

  • Even easier way than what I got. – ryekayo Oct 31 '14 at 15:15
0

The closest thing to a multithreaded process is the & which runs commands in the background.

So to use this command, you would do something like:

cp file location &
  • I want to be able to specify the number of parallel threads used. – leeand00 Oct 31 '14 at 15:09
  • You can probably stick this in a function and take in a parameter to accept ints to specify the number of times you want the command run..That'll be a bit of coding on your end though – ryekayo Oct 31 '14 at 15:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.