2

Does the mv command (temporary) duplicate the moved files when the target is an S3FS mount?

I have a VM that is reaching total space consumption. So, I'd like to move some files to a storage bucket (mounted using S3FS) to free up some space.

Since I'm currently using 88.7% of 28.90GB, which is approximately 25.70GB and the directory I plan to move is 8.2GB, I'm afraid to use 100% and get stuck.

mv /path/sourcefolder/* /path/destinationfolder/
2
  • Do you have the option of just moving a few files manually first, so you have some breathing room, and only then move *?
    – terdon
    Sep 15, 2023 at 10:29
  • If S3FS is just configured to save temporary files somewhere else (there might not be a "somewhere else" on this VM) I guess it doesn't matter what it does. Sep 15, 2023 at 11:06

2 Answers 2

6

s3fs uses local temporary storage to write objects to S3. -o max_dirty_data controls this amount and is set to 5 GB by default. When copying a larger file, e.g., 10 GB, s3fs will write the first 5 GB locally then write to S3. Afterwards it will write the second 5 GB to both then use a server-side copy to assemble the final object. Thus if you are worried about using too much local storage you should decrease this value. But doing so will trigger additional server-side copies, possibly slowing the overall write operation.

2
  • Thanks for your reply. Forgot to mention I'm talking about many "small" (up to 20/30 MB) files.
    – Sig
    Sep 15, 2023 at 15:37
  • By default s3fs only uses local temporary space when writing files and you will only observe this with large ones. If you enable -o use_cache it also stores data for future reads of files and should remove these when the file is closed. You can control the latter indirectly via -o ensure_diskfree. Sep 15, 2023 at 20:04
-1

mv does not duplicate the files. cp does.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .