Is it due to the buffering in memory? I did cp /network/large_file . & and I kept doing ls -lh to monitor progress, but the file didn't appear until it was about 400MB. Linux 2.6 kernel, Bash 4, 4GB swap, 16GB RAM.

  • You'll probably get better answers if you tell us how you're copying this file from the network. – rahmu Aug 30 '13 at 22:19
  • The file is on a CIFS share, mounted with mount.cifs to /mnt/network being copied to an EXT4 filesystem using the commands I listed. What else would you like to know? The file is ~ 260 GB. – Gregg Leventhal Aug 31 '13 at 1:49
  • Also The local hard disk is virtualized in VMware (vmdk format) and the backing datastore is a hardware RAID 5 with 5 disks using a PERC controller in a Dell R620 server. – Gregg Leventhal Aug 31 '13 at 1:58

This is likely to be dependent on your underlying filesystem, possible hdd buffers, raid controller buffers (if you are using raid), etc. When a file is copied, it first starts the stream in memory, notifying the disk there is a pending write operation - and the disk takes care of it from there. It's very common for a write to have "completed", but the file is actually still being written by the disk. Some filesystems prevent this "lag" such as ZFS and I believe BtrFS (this is known as copy-on-write). Other filesystems will report the copy/write operation is completed when it's not really -- which can lead to file corruption if power is lost, etc. ZFS and BtrFS don't report back to the OS that a file write operation is completed until it is fully complete -- preventing this situation.

To test this, try doing a copy of something largerish - such as a DVD iso of your favorite distro from one directory to another. The command line will return to the prompt in a short while, but if you immediately type sync and send the command, you'll notice it will hang the command line until the write operation is fully complete. (if you have disk activity lights, you'll also notice they are still blinking)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.