76

I had a problem (new to me) last week. I have a ext4 (Fedora 15) filesystem. The application that runs on the server suddenly stopped. I couldn't find the problem at first look.

df showed 50% available space. After searching for about an hour I saw a forum post where the guy used df -i. The option looks for inodes usage. The system was out of inodes, a simple problem that I didn't realize. The partition had only 3.2M inodes.

Now, my questions are: Can I make the system have more inodes? Should/can it be set when formatting the disk? With the 3.2M inodes, how many files could I have?

Improve this question
2

6 Answers 6

Reset to default
43

It seems that you have a lot more files than normal expectation.

I don't know whether there is a solution to change the inode table size dynamically. I'm afraid that you need to back-up your data, and create new filesystem, and restore your data.

To create new filesystem with such a huge inode table, you need to use '-N' option of mke2fs(8).

I'd recommend to use '-n' option first (which does not create the fs, but display the use-ful information) so that you could get the estimated number of inodes. Then if you need to, use '-N' to create your filesystem with a specific inode numbers.

Improve this answer
6
  • 12
    You can use mke2fs -i to specify the number of inodes. Its documentation indicates that “it is not possible to expand the number of inodes on a filesystem after it is created”.
    – Gilles 'SO- stop being evil'
    Dec 12, 2011 at 0:35
  • 2
    @piovisqui: each file consume on inode, which is a pointer in the filesystem. if the file is an hard-link to another file it has the same inode.
    – Hanan N.
    Dec 12, 2011 at 4:47
  • 9
    @Gilles The -i options specifies the size of the inode, not how many there are. The -N option sets the number inodes.
    – theillien
    Oct 23, 2014 at 20:12
  • 1
    The relation between inodes and file numbers isn't necessarily 1:1. The first inode contains a list of pointers to blocks where the file is stored. If the list of blocks can't fit within one inode, then the inode contains a list of pointers to inodes which list the blocks where the file is stored. If it doesn't fit there then it goes 3 sets of inodes deep for that list of blocks etc.
    – StuWhitby
    Apr 14, 2015 at 14:02
  • 5
    @StuWhitby That's not quite right. A single inode has several direct pointers, and a single, double, and triple indirect pointer. If the list of blocks can't fit in the direct pointers, then the single indirect pointer will point at a block of data (NOT another inode) that contains more pointers. If more pointers than that are needed, the double indirect pointer points at a block that contains single indirect pointers, and the triple indirect at a block with double indirect pointers. So a file does, in fact, just use one inode, regardless of size.
    – user125355
    Nov 30, 2017 at 14:13
18

With 3.2 million inodes, you can have 3.2 million files and directories, total (but multiple hardlinks to a file only use one inode).

Yes, it can be set when creating a filesystem on the partition. The options -T usage-type, -N number-of-inodes, or -i bytes-per-inode can all set the number of inodes. I generally use -i, after comparing the output of du -s and find | wc -l for a similar collection of files and allowing for some slack.

No, it can't be changed in-place on an existing filesystem. However:

  • If you're running LVM or the filesystem is on a SAN's LUN (either directly on the LUN, or as the last partition on the LUN), or you have empty space on the disk after the partition, you can grow the partition and then use resize2fs to expand the filesystem. This adds more inodes in proportion to the added space, roughly. If you want to avoid running out of inodes before space assuming that future files on average have about the same size, set a high enough reserved block percentage using tune2fs -m.
  • If you have enough space and can take the filesystem offline, then take it offline, create a new filesystem with more inodes, and copy all the files over.
  • If just a subset of the files are using a lot of the inodes and you have enough free space, create a filesystem on a loop device backed by a file on the filesystem, create a filesystem with more inodes (and maybe smaller blocks as well) on it, and move the offending directories into it. That's probably a performance hit and a maintenance hassle, but it is an alternative.
  • And of course, if you can delete a lot of unneeded files, that should help too.
Improve this answer
17

As another workaround I could suggest considering packing huge collections of files into an uncompressed(!) tar archive, and then using archivemount to mount it as a filesystem. A tar archive is better for sharing than a filesystem image and provides similar performance when backing up to a cloud or another storage.

If the collection is supposed to be read-only, squashfs may be an option, but it requires certain options enabled in the kernel, and xz compression is available for tar as well with the same performance.

Improve this answer
1
  • 3
    Nice suggestion.
    – piovisqui
    Nov 30, 2015 at 17:38
10

I have an alternate solutions for this situation. Lets say you have 1000 inodes in a partition of 10G. But due to inodes limit you are not suppose to use all space of partition. But in this solutions you will be able to use the remaining space of the partition without formatting it.

$ df -i  # see list ( I need just one free inode here so move just one file into other PARTITION)
/dev/part1  1000 999 1 99.9%     /data

$ dd if=/dev/zero of=/data/new_data
$ mkfs.ext4 /data/new_data
$ mkdir /data1
$ mount /data/new_data /data1

for permanent mounting

$ echo "/data/new_data /data1 ext4 defaults 0 1" >> /etc/fstab
Improve this answer
3
  • 3
    Welcome to U&L. I took the liberty to reformat your answer to the more usual representation of code here, inserting a prompt ($) to clearly distinguishing between commands and output (if they were only command, the prompt is normally left out). I also changed the SHOUTING in the bold phase emphasis, which is what I think you intended. You can roll back the changes if I misrepresented things
    – Anthon
    May 25, 2015 at 5:26
  • I think this solution has logic, but you need to manage the size when running dd.
    – piovisqui
    May 26, 2015 at 14:51
  • 3
    The details are wrong, you'd need to use a loop device, and maybe even unionfs depending on the application, but this is the only solution avoiding formatting and restoring from backup which is no fun when in a hurry with millions of files. There are circumstances where this could save the day !
    – medoc
    Jul 29, 2015 at 20:43
7

try du -s --inodes * 2>/dev/null |sort -g the cd into the last dir in output and repeat.

Full Disclosure: not all OS's support --inodes flag for du command (my Mac OS does not) but many Linux OS's do.

Improve this answer
5

Recently ran into this issue when using apt or aptitude upgrade.

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  5.1G  2.3G  70% /

df -i

Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 521497   2791  100% /

Issued command:

du /|sort -k1 -n

Revealed most of files were in subfolders for several kernel versions within:

/usr/src/linux-headers

Removed those subfolders and inode problem was fixed.

df -i

Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 104986 419302   21% /
Improve this answer
8
  • does "du /|sort -k1 -n" show inodes?
    – Orphans
    Jan 3, 2017 at 14:13
  • No. That was to sort the directories, showing which ones had the most files in them, folders which were consuming lots of inodes but less actual space usage: the situation of 30% free disk space yet 100% inode usage shown above.
    – kph0x1
    Jan 3, 2017 at 17:42
  • I honestly dont get how "du" shown how many files there are with any flag? Could you please explain more in detail?
    – Orphans
    Jan 4, 2017 at 17:57
  • No flags for the du command. Usage is for the root of file-system in above example, looking at space only. Output is piped to sort in order to show which directories contained the most files. There isn't a count done in above example for files, the 'how many files' portion of your question. Kernel source ones were the culprit shown in the du output though; e.g., many small files, sub-folders from past compilations, the very thing ideally suited for removal in order to free up inodes. There still remains a manual, human review of the du output, /usr/src/linux-headers was then obvious.
    – kph0x1
    Jan 9, 2017 at 15:52
  • 3
    du shows ONLY bytes - not files. And you are piping the outout only from the du command into sort. So how does sort -k1 -n sort the output in that way you proposed? The only thing I can see is that "du /|sort -k1 -n" only sort every row based on the size in bytes. Nothing else
    – Orphans
    Jan 9, 2017 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.