I want to have a 24TB disk prepared to be able to contain huge number of dirs and files in a single directory (Please don't tel me to change this strategy, it is an structure used by a software which is a black box for us so we can't change this approach). As I have fully researched, ext4 filesystem has the capability to store billions of file in one directory, but it should be prepared using some special flags and parameters. This is what I have used based on my research:
sudo mke2fs -T news /dev/sdb1
sudo tune2fs -O dir_index /dev/sdb1
sudo tune2fs -O large_dir /dev/sdb1
sudo tune2fs -O dir_nlink /dev/sdb1
sudo mkdir /hdd
sudo gedit /etc/fstab
- add following to the end of the file:
/dev/sdb1 /hdd ext4 defaults,noatime 0 0
sudo mount /hdd
To test the structure, I have prepared bash scripts which create dirs and files in a single directory. Something like this:
for ((i = 1000000; i <= 200000000; i++))
sudo mkdir "/hdd/largedir/$i" -p
sudo cp "sample-file.jpg" "/hdd/largedir/$i"
if (( $i % 1000 == 0 ));
echo "$i created";
After hours of work, when I checked the system it was printing this error:
Structure needs cleaning
On my tests, it prints this error not for all file and dirs. For example I can create a dir named "10000" but cannot create a dir named "1000". I also changed hash algorithm using this command:
sudo tune2fs -E "hash_alg=tea" /dev/sdb1
and rebooted and remounted the system, but the problem still exists. Do anyone knows what is the problem and why this situation has been occurred on the filesystem? Does ext4 filesystem is not strong enough for having such large structure? I have read on some pages about using xfs instead of ext4 for large number of files. Is it actually true?
You see that during file operation, no power loss, neither no crashing have been occurred on the system. I not expected such a behavior when everything was ok.
-- EDITED for more information: --
The inode information for the disk is as follows:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb1 421216256 183643803 237572453 44% /hdd
The space information is as follows:
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 26T 3.1T 21T 13% /hdd
I have checked the filesystem and it is ext4 (as shown in various tools). For example gparted shows the partition filesystem as ext4. About the features, none of the features above were enabled by default on my ubuntu 18.04 LTS. I have reached several errors during my old tests and finally reached to this.