0

I am using a xfs file system at work for storing image processing data. Currently, it has around 8.8T of free space.

/dev/sdh1             106T   97T  8.8T  92% 

While there are plans to move some of the data to tape and make room, it won't happen until next week. Currently, I keep running into "No space left on device" error quite regularly.

Usually the images trasferred are around 128mb in size and they are around 100-500 of them at time.

Is there anything specific to the file system that is making these ~8TB of free space unusable?

On my end, I was able to verify that I can use atleast 8TB of this space using the command fallocate to create really large files of around a TB.

What I am missing? Are there any obvious file-system level checks that I need to do?

For your reference, here is the output of the command xfs_info for the filesystem.

meta-data=/dev/sdh1              isize=256    agcount=106, agsize=268435455 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=28319810304, imaxpct=1
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

In order to reproduce the same error, I wrote a simple shell script that creates a large number of files(10k) small files (1M in size) and it fails with the following error:

fallocate: temfile-7464: open failed: No space left on device

Here is the output of df -i before the script is run

/dev/sdh1             4531169600  648793  4530520807    1% /jumbo/K2LEGINON

And after

/dev/sdh1             4531169600  656256  4530513344    1% /jumbo/K2LEGINON

It failed after creating around ~7500 files. Which amount to ~ 7.3G.

  • What is giving you the lack of space errors? A mv command in Bash? A web front-end to the image processing application? – Wildcard Aug 29 '16 at 5:15
  • The images are usually trasferred from another local computer where the the parition is used via nfs/samba. – feverDream Aug 29 '16 at 5:27
  • What does df -i show? – Julie Pelletier Aug 29 '16 at 5:34
  • I update the question with that information – feverDream Aug 29 '16 at 6:01
  • 2
    This is often a problem on 32-bit inode file systems serverfault.com/questions/696064/… – Mark Plotnick Aug 29 '16 at 7:04
0

Thanks to responses in the comments, I was able to identify the problem as XFS filesystem related.

Essentially, the filesystem was running out free space for creating inodes. The XFS filesystem seems to be using the first one Terabyte of the disk to do so.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.