ext4 Ubuntu 18.04 File System Error "Strcuture needs cleaning"

Question

I want to have a 24TB disk prepared to be able to contain huge number of dirs and files in a single directory (Please don't tel me to change this strategy, it is an structure used by a software which is a black box for us so we can't change this approach). As I have fully researched, ext4 filesystem has the capability to store billions of file in one directory, but it should be prepared using some special flags and parameters. This is what I have used based on my research:

sudo mke2fs -T news /dev/sdb1
sudo tune2fs -O dir_index /dev/sdb1
sudo tune2fs -O large_dir /dev/sdb1
sudo tune2fs -O dir_nlink /dev/sdb1


sudo mkdir /hdd
sudo gedit /etc/fstab

- add following to the end of the file:
/dev/sdb1    /hdd    ext4    defaults,noatime    0    0

sudo mount /hdd

To test the structure, I have prepared bash scripts which create dirs and files in a single directory. Something like this:

for ((i = 1000000; i <= 200000000; i++))
do
  sudo mkdir "/hdd/largedir/$i" -p
  sudo cp "sample-file.jpg" "/hdd/largedir/$i"
  if (( $i % 1000 == 0 ));
  then
    echo "$i created";
  fi;
done

After hours of work, when I checked the system it was printing this error:

Structure needs cleaning

On my tests, it prints this error not for all file and dirs. For example I can create a dir named "10000" but cannot create a dir named "1000". I also changed hash algorithm using this command:

sudo tune2fs -E "hash_alg=tea" /dev/sdb1

and rebooted and remounted the system, but the problem still exists. Do anyone knows what is the problem and why this situation has been occurred on the filesystem? Does ext4 filesystem is not strong enough for having such large structure? I have read on some pages about using xfs instead of ext4 for large number of files. Is it actually true?

You see that during file operation, no power loss, neither no crashing have been occurred on the system. I not expected such a behavior when everything was ok.

-- EDITED for more information: --

The inode information for the disk is as follows:

Filesystem        Inodes     IUsed     IFree IUse% Mounted on
/dev/sdb1      421216256 183643803 237572453   44% /hdd

The space information is as follows:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        26T  3.1T   21T  13% /hdd

I have checked the filesystem and it is ext4 (as shown in various tools). For example gparted shows the partition filesystem as ext4. About the features, none of the features above were enabled by default on my ubuntu 18.04 LTS. I have reached several errors during my old tests and finally reached to this.

Answer 1

With the command sudo mke2fs -T news /dev/sdb1 you aren't necessarily making an ext4 filesystem, but an ext2 filesystem.

Assuming that your Ubuntu 18.04 has essentially the same /etc/mke2fs.conf file as my Debian 10, then dir_index is already enabled as the current base feature set for all ext2/3/4 filesystems created using modern mke2fs. And dir_nlink is enabled by default for the ext4 filesystem type.

The mke2fs.conf(5) man page says: "if both the user and the mke2fs.conf file do not specify a default filesystem type, mke2fs will use a default filesystem type of ext3 if a journal was requested via a command-line option, or ext2 if not."

According to the /etc/mke2fs.conf file, the -T news option only specifies the inode_ratio = 4096 option, nothing else. So unless you use the mkfs.ext4 form instead of the plain mke2fs, you might be getting an ext2 filesystem that is tailored for files with average sizes of 4 kB or less.

Debian does not have the fs_type = specified in the [defaults] section of mke2fs.conf, and you don't include the -j option for your mke2fs command, so if Ubuntu's mke2fs.conf is the same as Debian's (as it usually is), your command may have given you an 24 TB ext2 filesystem, which is something that probably hasn't been tested particularly well by anyone.

The ext4(5) man page says the 64bit filesystem feature will be set automatically as needed, which might explain why the tools did not report any errors. It also says the dir_index feature is ignored by ext2 filesystems.

From past experience with a multi-terabyte ext3 filesystems, I would have expected huge filesystem creation and checking times. For your use case, the presence or absence of the dir_index feature might make or break the application performance for you.

Could you please edit the actual tune2fs -l /dev/sdb1 output to your original question, to eliminate the need for us to guess what may or may not have happened?

"Structure needs cleaning" seems to be the default text corresponding to the EUCLEAN kernel error code, which indicates the filesystem is corrupted and a filesystem check is needed. On a filesystem of this size, this is going to take quite a bit of time and RAM. And of course, the filesystem needs to be unmounted while it's being checked.

Answer 2

Actually after a lot of tests, I reached to the point that seems ext4 does not have the ability to do such a thing (holding billions of files in one dir). And following my researches on how to do this in linux, the results (in fact also my practical tests) shows that for such a scenario, I should use XFS instead of ext4, which is really built to accomplish this task.