9

While moving a big chunk of data between two external USB drives, I notice my laptop is slowed down. It was my understanding that the files are not written to any intermediate location (such as /tmp or similar) unless there is a shortage of free RAM. Am I wrong?

  • 2
    The slowdown may be caused by the copied files getting cached and pushing more useful things out of RAM. It seems to be a common problem on Linux. – user253751 Oct 21 at 10:19
  • Some system monitoring tools can tell you disk usage per-disk, so you can check if the internal disk is being (heavily) used while copying: serverfault.com/questions/9428/… – golimar Oct 21 at 16:25
17

If you have a copy such as this, or its GUI equivalent,

cp -a /media/external/disk1/. /media/external/disk2/

the data is read from the first disk's filesystem and written directly to the second. There is no intermediate write to another storage location. If you are seeing slow speeds it may be that the two disks are sharing the same USB controller and contending for access to the bus.

Anything more than that and you will have to provide further details, such as the make/model of computer, its bus topology, etc.

| improve this answer | |
  • If I copy them using a graphical interface (i.e. Ctrl+C Ctrl+V in Nautilus), the answer is the same. Right? – Mephisto Oct 20 at 17:31
  • 1
    @Mephisto yes, it’s the same. – Stephen Kitt Oct 20 at 18:40
  • Can't parts of it still land somewhere on your internal hdd? Data is not written byte by byte, but in chunks, and those are held temporarily in RAM. And it might theoretically happen that some of that RAM gets paged if the operation takes long enough and some other ram-hungry process gets prioritized.. – vsz Oct 21 at 7:53
  • 2
    @vsz the OP discounted swapping when they wrote, "unless there is a shortage of free RAM". However, to address your question, it might be theoretically possible for block data to land in swap. On the other hand depending on the type of copy it could, also theoretically, be a direct copy between the two devices without the CPU even being involved. I don't know that level of detail – roaima Oct 21 at 8:17
  • @vsz Any buffers used in the copy process would ipso facto be very actively accessed memory. Anything causing these to be swapped would have to be really RAM-hungry – Hagen von Eitzen Oct 21 at 17:10
4

There is no concept of internal and external: The operating system knows not of the box, and what is in or out of it. The only file-system that is special (but only a little bit) is the root file-system, and this can be mounted read-only. You can still do almost anything with a read-only root file-system (but not change the content of the root file-system).

| improve this answer | |
  • The concept of internal and external is very useful here (assuming the root fs is internal). A common cause of computers being slow is if the device behind the root fs is too busy doing IO. It means the OS can't access the data of the running binaries etc. without waiting for the drive. This is a very common problem on cheap windows 10 computers with a slow hard drive and not enough ram. I've also seen it on Ubuntu while copying large files to the root fs. The ability to rule out this as a cause is very useful. Otherwise I'd have posted an answer suggesting it as the cause. – HEGX64 Oct 21 at 3:05
  • 3
    It might be important for the user where a disk is and how it is connected, for the Linux operating system however they are all „filesystems“, therefore copy from one „external“ disk to another „externel“ will not be buffered on a „internal“ disk. If copying slows down the whole system, then it is due to high driver cpu or interrupt load. – eckes Oct 21 at 4:39
  • 1
    @HEGX64 the distinction is not internal or external. if is root or not-root. Or it is same device or different device. – ctrl-alt-delor Oct 21 at 9:07
1

TL,DR: it's probably the cache.

Copying or moving data between two removable drives doesn't store the data on an internal drive. The copying program reads a chunk of data from the source drive, writes it to the target drive, and repeats with the next chunk of data. A sophisticated program may try to make the copy faster, at the expense of heavier load during the copy, by copying multiple chunks in parallel.

Accessing drives takes some CPU time, both for the filesystem and for the actual drive access. The cost of the filesystem management is small if you have a lot of small files and negligible for large files. The cost of drive access depends on the quality of the USB controller and of the driver. I think USB controllers and their Linux drivers generally use DMA reasonably efficiently so there's not much CPU cost here either, but I know very little about USB and may be wrong here.

The data that is being copied is cached like any other data that is read from a disk. If you're copying a large amount of data (relative to the size of your RAM), the data that you're copying will displace frequently-accessed files and will even cause part of the memory of applications to be moved to swap. This can cause a significant performance hit. The system doesn't know that the data that it's reading from the source disk is only meant to be written out immediately, and doesn't need to be kept in RAM, so it tends not to make the best decision in terms of cache allocation.

When you copy a large amount of data on Linux, try using nocache (available as a package in major distributions). This is a command line utility that runs a command in a mode where the data that it reads will mostly not go into the cache. For example:

nocache mv /media/source/dir /media/target/dir

Using the nocache prefix can work on a GUI application, but only if the application starts normally. Some desktop environments arrange to start applications via a central process (e.g. KDE with kdeinit), or may open a window in an existing process, and in these cases nocache won't have any effect.

If you're concerned about privacy, it's unlikely, but possible that some of the data could end up in the swap space. It's unlikely because the copying program only keeps each chunk of data in memory for a small amount of time, and swap is used in priority for parts of the memory that haven't been accessed for a long time.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.