Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've been having a fairly serious issue on a high traffic web server. PHP pages are slowing down considerably, and it only seems to be an issue on pages where sessions are accessed, or a certain table within a database is being referenced. the '/var/log/messages' log file, I see hundreds of thousands of the following error: 'kernel: EXT4-fs warning (device dm-0): ext4_dx_add_entry: Directory index full!'

I suspect there is a bottleneck in '/var/lib/php/sessions' because I cannot open the folder in Filezilla, and cannot count the number of files/sub-directories with grep. While it is quite possibly a case of hard drive corruption, I'd like to verify a hunch of mine first by checking the number of files inside of this directory.

How would you go about finding the number of files within a folder without actually counting the files in said folder?

share|improve this question

migrated from stackoverflow.com Aug 15 '13 at 17:22

This question came from our site for professional and enthusiast programmers.

    
In PHP or in general? –  Gordon Aug 15 '13 at 17:18
    
In general. Assume you have ssh access to the machine. –  Pangamma Aug 15 '13 at 17:20
    
What does df -i say? –  Gilles Aug 15 '13 at 22:39

3 Answers 3

up vote 3 down vote accepted

The size of the directory (as seen with ls -ld /var/lib/php/sessions) can give an indication. If it's small, there aren't many files. If it's large, there may be many entries in there, or there may have been many in the past.

Listing the content, as long as you don't stat individual files, shouldn't take a lot much longer than reading a file the same size.

What might happen is that you have an alias for ls that does ls -F or ls --color. Those options cause an lstat system call to be performed on every file to see for instance if they are a file or directory.

You'll also want to make sure that you list dot files and that you leave the file list unsorted. For that, run:

command ls -f /var/lib/php/sessions | wc -l

Provided not too many filenames have newline characters, that should give you a good estimate.

$ ls -lhd 1
drwxr-xr-x 2 chazelas chazelas 69M Aug 15 20:02 1/
$ time ls -f 1 | wc -l
3218992
ls -f 1  0.68s user 1.20s system 99% cpu 1.881 total
wc -l  0.00s user 0.18s system 9% cpu 1.880 total
$ time ls -F 1 | wc -l
<still running...>

You can also deduce the number of files there by subtracting the number of unique files elsewhere in the file system from the number of used inodes in the output of df -i.

For instance, if the file system is mounted on /var, with GNU find:

find /var -xdev -path /var/lib/php/sessions -prune -o \
  -printf '%i\n' | sort -u | wc -l

To find the number of files not in /var/lib/php/sessions. If you subtract that to the IUsed field in the output of df -i /var, you'll get an approximation (because some special inodes are not linked to any directory in a typical ext file system) of the number of files linked to /var/lib/php/sessions that are not otherwise linked anywhere else (note that /var/lib/php/sessions could very well contain one billion entries for the same file (well actually the maximum number of links on a file is going to be much lower than that on most filesystems), so that method is not fool-proof).

Note that if reading the directory content should be relatively fast, removing files can be painfully slow.

rm -r, when removing files, first lists the directory content, and then calls unlink() for every file. And for every file, the system has to lookup the file in that huge directory, which if it's not hashed can be very expensive.

share|improve this answer

I guess you could use ls -1 /var/lib/php/sessions | wc -l over SSH

Note that it is ls -(one) and ws -(letter l)

share|improve this answer
    
Tried it. It has been going for about a minute now. No results yet. Any ideas? –  Pangamma Aug 15 '13 at 17:34
1  
@Pangamma an extremely large directory is going to take a while to read. –  jordanm Aug 15 '13 at 17:43
    
It shouldn't take more time that a simple ls... Can you do ls /var/lib/php/sessions ? –  Mash Aug 15 '13 at 17:44
    
@mash I have seen directories with hundreds of thousands of files take 10 minutes to ls. –  jordanm Aug 15 '13 at 17:47
1  
The portable and canonical way to do that is with ls -f –  Stéphane Chazelas Aug 15 '13 at 18:49

In ext2/3/4 you can make some approximation based on the size of the directory itself. This is not very accurate, but allows an educated guess without "counting".

A directory starts at 4096 bytes by default. As it gets more entries, it grows in size (but will never shrink again). After some testing it appears that each entry takes 12 bytes, and grows by 4096 when it reaches the limit. This means that if the current directory size is 16384, you have had between 1,024 and 1,365 entries at one time.

There is also a trick to viewing files in these directories immediately. When you run ls, the entire directory is read and then sorted. You can disable the sorting using ls -1U.

share|improve this answer
1  
Well a filename that is 200 characters long will typically use more that 12 bytes. –  Stéphane Chazelas Aug 15 '13 at 18:52
    
@StephaneChazelas good note, it is a pretty inaccurate way to calculate. –  jordanm Aug 15 '13 at 19:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.