11

I have a server which stores a lot of data, I keep many nested directories with millions of files in them.

While trying to move one big directory (~700GB) from one disk to another I get an error:

mv /media/storage3/dir /media/storage4/

"No space left on device"

Also, trying to create a small txt file on the disk does not work:

Error opening file '/media/storage4/Untitled Document': No space left on device

I checked many options online, none worked.

df -h output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg7        77G  9.2G   64G  13% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev             32G  4.0K   32G   1% /dev
tmpfs           6.3G  1.4M  6.3G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none             32G   76K   32G   1% /run/shm
none            100M   48K  100M   1% /run/user
/dev/sdg6        19G  6.7G   11G  39% /home
/dev/sdb1       440G  278G  140G  67% /var
/dev/sda1       2.8T  2.0T  764G  73% /media/storage1
/dev/sdh1       2.8T  1.6T  1.3T  56% /media/storage2
/dev/sdi1       2.8T  1.7T  1.2T  60% /media/storage3
/dev/sdf1       2.8T  876G  1.9T  32% /media/storage4

df -i output:

Filesystem         Inodes    IUsed      IFree IUse% Mounted on
/dev/sdg7         5079040   514215    4564825   11% /
none              8242449        2    8242447    1% /sys/fs/cgroup
udev              8239750      611    8239139    1% /dev
tmpfs             8242449      696    8241753    1% /run
none              8242449        5    8242444    1% /run/lock
none              8242449        4    8242445    1% /run/shm
none              8242449       28    8242421    1% /run/user
/dev/sdg6         1220608   218613    1001995   18% /home
/dev/sdb1        29310976 12863877   16447099   44% /var
/dev/sda1       858436804 41630853  816805951    5% /media/storage1
/dev/sdh1      1356948436 38728057 1318220379    3% /media/storage2
/dev/sdi1      1217505624 34748869 1182756755    3% /media/storage3
/dev/sdf1      2048962648 36308921 2012653727    2% /media/storage4

Reading online, people said similar problems happen because of access to a huge amount of files and that setting fs.inotify.max_user_watches to a high amount might solve it, I increased it from 8192 to 1000000 but it did not help.

  • Have you tried lsof +L1? man lsof: A specification of the form +L1 will select open files that have been unlinked. – A.B. Mar 26 '15 at 12:57
  • There is no reason for mv to use the inotify mechanism. Have a look here : some processes may still be using unlinked files from the destination filesystem. – John WH Smith Mar 26 '15 at 12:57
  • Deleted files would still show up in the usage given by df... – Stephen Kitt Mar 26 '15 at 13:00
  • @StephenKitt No. Unlinked files which are held by the file subsystem due to inode referencing will. Those which got completely unused and unlinked ("deleted") do not show up. See here. – John WH Smith Mar 26 '15 at 13:37
  • 1
    Check ntfsinfo -m /dev/DEV for clues. It's part of ntfs-3g. Also note that there can only be up to 2^32 files on a single NTFS volume. I wouldn't trust any native linux/unix tool to report NTFS inodes count precisely. – mikky Jun 3 '15 at 16:18
1

Run the following:

lsof -s | sort -nrk 7 | head

You'll see output like so:

firefox    2997                     j   52rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j   50rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j    3rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j   39rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j   31ur     REG              252,0 10485760    5505182 /some/path

Note; the above is just an example you'll need to use your judgement to find the space usage.

About output:

  • Column 7 is the size of the space in use.
  • Column 4 is the file descriptor.

You can remove the content at the file-descriptor to gain back that space:

cat /dev/null > /proc/2997/fd/50

Obviously this will destroy all data on that file descriptor.

  • If the problems occurs again, I will give it a try – PeterN Aug 25 '15 at 17:54
0

You could try a tar to copy all the data, then remove it afterward:

cd /media/storage3/; tar cfp - ./dir | ( cd /media/storage4; tar xfp - )

This will preserve all ownership, permissions and links

  • No copy worked.. Not only that, creating a simple empty file returned no space left exception – PeterN Apr 23 '15 at 8:05
0

I would have used rsync -avrz storage3 storage4 and then deleted the files afterwards, but I got another question why do you have those shares using NTFS filesystem if you are using Linux (Ubuntu) then I would have gone for ext4 or ReiserFS, ButterFS. IF those shares are located on a Windows box, please try and move those files from Windows to Windows using robocopy, which is excellent in this regards. Robocopy works mostly like rsync.

man page for rsync: http://linux.die.net/man/1/rsync

help page for robocopy: https://technet.microsoft.com/en-us/library/cc733145.aspx

support page for 3g-ntfs drivers for Linux: www.tuxera.com/community/open-source-ntfs-3g/

FAQ for 3g-ntfs: www.tuxera.com/community/ntfs-3g-faq/

I suspect something should be fixed inside the NTFS filesystem drivers you are using in your Ubuntu installation, which distribution of NTFS are you using, how did you mount the disks, are they local attached to the ubuntu box you are sitting on or are they mounted using iSCSI from a SAN?

There is more to this troubleshooting than, just "I cannot move my files they are located on NTFS and I'm running ubuntu to move them."

  • As mentioned, no copy worked, i tried all sorts of rsync commands. But, even creating an empty file on storage4 returned an exception. \n The reason for using NTFS is because we have other servers, which do run windows, and we wanted to allow for easy transfer of disks from one server to another. In retrospect, I wouldn't have used NTFS because of all the problems it brings with it, Ubuntu is very unstable with an NTFS mount. \n The disks are connected via SATA cable directly to the motherboard on the computer, no SAN or anything. It's a direct mount on the linux system. – PeterN Apr 23 '15 at 8:06
  • I would advice you to two things, when using MS technologies go pure MS, use a Microsoft Windows Fileserver if you need to have NTFS, else go with ext4 or other Linux filesystem and share the disk with Smb. – OMG-1 Apr 23 '15 at 14:43
  • Save yourself some problems and never try to combine windows and linux. No amount of help will save you in the long run, eventually this will always end in tears. It does seem like a problem with inode or file handles, but when you put NTFS in the mix... time bomb. If your environment needs interaction of linux and NTFS, put the NTFS on a windows server and do the interaction that way. Not trying to combine onto the same system. – Baazigar May 19 '15 at 18:42

protected by Community May 21 '15 at 17:10

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.