1

This question resulted from a question of mine on stackexchange.

I mount a windows share with mount -t cifs -o username=username,password=password,rw,nounix,iocharset=utf8,file_mode=0777,dir_mode=0777 //192.168.1.120/storage /mnt/storage, then have scripts operate on the mountpoint on a debian machine.

The mountpoint /mnt/storage will contain a very rapidly growing number of files which are moved to a subdirectory in batches, where they are processed.

My problem is that the moving is relatively slow and I fear that this is because I'm unable to just alter the filetable (simply change the information where the file is on the hdd).

I'm currently using shutil.move(src,dst) in python, but also considered using os.rename(src,dst) or a subprocess using mv.

Is my fear correct - if so is there a way to mount more efficiently?

edit: I just went over the documentation of shutil.move() again and read this:

If the destination is on the current filesystem, then os.rename() is used.
Otherwise, src is copied (using shutil.copy2()) to dst and then removed.

This sounds like it could be an issue, if it is not "on the current filesystem", how do I know if I am "on the current filesystem"

edit2: In case someone's interested in what I found out, I'll copy&paste me edit from stackoverflow here as well..

I wrote a script to test the speed differences between the various methods of moving. First created 1x5GB (dd if=/dev/urandom of=/mnt/storage/source/test.file bs=100M count=50), then with 100x5MB (for i in {1..100}; do dd if=/dev/urandom of=/mnt/storage/source/file$i bs=1M count=5) and finally with 10000x5kB (for i in {1..100000}; do dd if=/dev/urandom of=/mnt/storage/source/file$i bs=1k count=5)

from shutil import move
from os import rename
from datetime import datetime
import subprocess
import os

print("Subprocess mv: for every file in directory..")
s = datetime.now()
for f in os.listdir("/mnt/storage/source/"):
    try:
        subprocess.call(["mv /mnt/storage/source/"+str(f)+" /mnt/storage/mv"],shell=True)
    except Exception as e:
        print(str(e))
e = datetime.now()
print("took {}".format(e-s)+"\n")

print("Subprocessmv : directory/*..")
s = datetime.now()
try:
    subprocess.call(["mv /mnt/storage/mv/* /mnt/storage/mvf"],shell=True)
except Exception as e:
    print(str(e))
e = datetime.now()
print("took {}".format(e-s)+"\n")


print("shutil.move: for every file file in directory..")
s = datetime.now()
for f in os.listdir("/mnt/storage/mvf/"):
    try:    
        move("/mnt/storage/mvf/"+str(f),"/mnt/storage/move")
    except Exception as e:
        print(str(e))
e = datetime.now()
print("took {}".format(e-s)+"\n")

print("os.rename: for every file in directory..")
s = datetime.now()
for f in os.listdir("/mnt/storage/move/"):
    try:
        rename("/mnt/storage/move/"+str(f),"/mnt/storage/rename/"+str(f))
    except Exception as e:
        print(str(e))
e = datetime.now()
print("took {}".format(e-s)+"\n")


if os.path.isdir("/mnt/storage/rename_new"):
    rmtree('/mnt/storage/rename_new')
print("os.rename & os.mkdir: rename source dir to destination & make new source dir..")
s = datetime.now()
rename("/mnt/storage/rename/","/mnt/storage/rename_new")
os.mkdir("/mnt/storage/rename/")
e = datetime.now()
print("took {}".format(e-s)+"\n")

Which revealed that there's not that much of a difference.. The 5GB file was moved really fast, which tells me that the moving, by altering the file table works. Here are the results of the 10000*5kB files (It felt like the results depend on the current network workload. e.g. the first mv test took 2m 28s, than later with the same files 3m 22s, also was os.rename() the fastest method most of the times..):

Subprocess mv: for every file in directory..
took 0:02:47.665174

Subprocessmv : directory/*..
took 0:01:40.087872

shutil.move: for every file file in directory..
took 0:01:48.454184

os.rename: for every file in directory..
rename took 0:02:05.597933

os.rename & os.mkdir: rename source dir to destination & make new source dir..
took 0:00:00.005704

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.