No space left on device although there is space available

Question

I can't create directory,

$ mkdir 2groups2
mkdir: cannot create directory ‘2groups2’: No space left on device

I can't even change directory

cd <TAB>

causes this

$ cd -bash: cannot create temp file for here-document: No space left on device

Simultaneously, df reports more than 9Gb:

$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
udev            31428656        0  31428656   0% /dev
tmpfs            6288016     9080   6278936   1% /run
/dev/xvda1      60923900 51409376   9498140  85% /
tmpfs           31440072   276252  31163820   1% /dev/shm
tmpfs               5120        0      5120   0% /run/lock
tmpfs           31440072        0  31440072   0% /sys/fs/cgroup
tmpfs            6288016        0   6288016   0% /run/user/1000

How to explain? And what to do?

WOW

df -i showed zero:

$ df -i
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
udev           7857164     397 7856767    1% /dev
tmpfs          7860018     488 7859530    1% /run
/dev/xvda1     7680000 7680000       0  100% /
tmpfs          7860018       1 7860017    1% /dev/shm
tmpfs          7860018       3 7860015    1% /run/lock
tmpfs          7860018      16 7860002    1% /sys/fs/cgroup
tmpfs          7860018       4 7860014    1% /run/user/1000

what does it mean?

Answer 1

Inodes are references to file contents within the disk. I like to think of them like file headers - they tell the operating system some metadata of the file along with the location of the actual data the file contains.

In a classic Unix filesystem design, every filesystem, in addition to having a fixed amount of data space, has a fixed amount if inode space. If either one of these becomes full, new files will be unable to be created on that partition. Here's a really good page that explains even more about inodes and how they are useful: http://www.grymoire.com/Unix/Inodes.html

An example of an inode's use is pointing two different files to the same data, also known as hardlinking. This saves data space in the partition at the cost of inode space.

In Linux's ext2/ext3/ext4 filesystem family, the ratio of inodes per unit of disk space can be specified at filesystem creation time and is then fixed for the rest of the life of that filesystem. If you run out of inodes, you can get more of them by expanding the total size of the filesystem, but this may be inefficient.

More modern filesystem designs like XFS will be able to create new inodes dynamically on-demand. There may or may not still be some restrictions on how far this mechanism can stretch, so you should make sure you're aware of the limitations of the filesystem type(s) you're using.