4

Lately I've been using USB2 flash drives to transfer a lot of data to my company.

When copying the Data to the flash drive, I noticed that the data is not written directly to the drive. It wrote with 200MB/sec to cache(?). I then umount or sync the drive and it takes hours to actually write the data to the drive.

I also tried using cp instead of Nautilus, but it had the same behavior. dd is not an option for copying normal files.

Is there any way to make Linux write the data directly without caching when writing on USB storage? Or maybe reduce caching so that I don't have to wait for hours when unmounting the drive?

What's annoying me most is that the progress bar finishes copying and I then have to wait a unspecified time for sync to write the data.

I'm on a Fedora 25 machine.

[Update] Tried to clarify my question.

[Update 2] Found a similar question without answer:Reduce cache size of flash storage devices

Improve this question
6
  • Maybe calling sync after cp can be useful? You'd wait for that to finish.
    – Paul Stelian
    Jul 7, 2017 at 12:10
  • I've tried with sync. It's the same behavior as with umount, as it writes the data after the copy process. It is really annoying when you see the progress bar finished and then have to wait a undefined time before you can remove the flash drive. I don't mind waiting for a while when writing >10GB of data to a USB2 device. It's just annoying not to know how long it will take and when it really is finished. The only thing I like about windows where the caching is less of a problem.
    – Gerald Zehetner
    Jul 7, 2017 at 19:37
  • There is an answer here, which affects the entire system not just removable devices, but a number of desktop users have found it useful: unix.stackexchange.com/questions/292024/…
    – sourcejedi
    Jan 21, 2019 at 11:08
  • Although, some of the claims in the linked LWN.net article are very suspicious IMO. unix.stackexchange.com/questions/480399/…
    – sourcejedi
    Jan 21, 2019 at 11:10
  • I too find it annoying. however, i do watch the progress after issuing <code>umount</code> with <code>watch -n 5 iostat /dev/your device </code>'
    – Ashutosh Gupta
    Dec 30, 2020 at 12:58

1 Answer 1

Reset to default
-4

You could dry the direct dd method. Sounds like it's time to reformat your drive with: mk2fs. -fs-of-your-choice.

man mkf2s

The dd route is simple.

Run once before you pulug in your drive. lsblk

And once after.

Notice the output of your device.

Then to copy the files you can create a new device like so: dd if=/dev/zero of=~/my-files.dd.img bs=1G mkfs.ext4 ~/my-files.dd.img losetup /dev/loop0 ~/my-files.dd.img

Note If you’d like this to be remounted after a machine reboot then add the above line to the rc.local file. vi /etc/rc.local

Mount the device with: mkdir /mnt/amazing mount -o loop=/dev/loop0 ~/my-files.dd.img /mnt/mymountpoint

To check the file has been mounted you can use the df command: df -h | grep mymountpoint /dev/loop0 976M 1.3M 924M 1% /mnt/mymountpoint where `bs=` a value greater than the output of: `ls -lh ./your-file-here`

When you're done unmount the devices: umount /mnt/mymountpoint losetup -d /dev/loop0

Haven't tested this but I suppose this is how it should work.

Cheers rivanov

Improve this answer
7
  • 2
    Hi. I don't really understand your answer. Fist of all I don't know why i should create a image file? I did reformat my usb-flash-drive many times because of the cache write the filesystem got corrupted. I even zeroed the first 1024 blocks of the device because it had a ISO written on it and the ISO Label wouldn't go away with simply reformatting. And writing on the device with dd without oflag=direct is also cached.
    – Gerald Zehetner
    Jul 5, 2017 at 19:57
  • try using wipefs /dev/yourdev
    – rivanov
    Jul 6, 2017 at 5:35
  • 1
    Writing with dd won't bypass the OS cache ...
    – dirkt
    Jul 6, 2017 at 7:33
  • @dirkt perhaps I did not understand the OPs question and rushed to answer.
    – rivanov
    Jul 6, 2017 at 14:51
  • 1
    He wants to monitor the completeness of writing to the USB flash drive. His idea was to disable buffering, so cp only returns when all data is really written to USB flash, and not when it has been buffered. Or one could use pv etc. The problem with that is of course that disabling buffering will cause duplicate writes for inodes/FAT. Making an image with dd, and mounting it on loop etc. has nothing to do with this question. In your place, I'd delete this answer ...
    – dirkt
    Jul 6, 2017 at 15:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .