1

Back in 2011 we had a peculiar problem.

We noticed that the processing tool for one of our clients was taking a long time processing ~20000 p/hr. Since we needed to process upwards 3 million files a month possibly in a 3 day span this was a problem for us.

We also noticed that this was not a CPU utilization issue which was around 10 percent. This tool used to write the files to a folder named by the area short code(e.g. DEL for Delhi). A folder could eventually hold anywhere between 5000 to 200000 files once the processing job was over.

We also observed that while files were being written to a folder, if we removed the files that were previously written to that folder(when > ~10000) into a subfolder, that sped up the job considerably.

Eventually the issue was solved when our tool vendor modified the tool to write files to folders named by zipcode within the folder with name of area code(DEL/110012 and so on). Each of these folders now contained to a maximum of approx 2000 files.

Edit 1: The OS used was HP-UX. Edit 2: File size averaged in the range 25-50 KB.

So why does it take longer to add new files to a directory that has a large number of files in it?

  • Should we guess what was the file system ? – Kiwy May 1 '18 at 8:59
  • Sorry not aware of that but the OS was HP-UX. – SidK May 1 '18 at 9:01
  • 1
    Then your question is a bit broad and opinion base, it will depend on how hpux has been setup at this time. – Kiwy May 1 '18 at 9:13
  • I was wondering if the issue was due to adding the entry corresponding to the file in the directory. As the number of entries increased adding the entry was somehow timetaking(thereby when the files were moved to a subfolder, it became easier to add entries to the directory again) – SidK May 1 '18 at 9:18
  • 1
    @Kiwy, it may lack enough details for people who don't know what would HP-UX use, but that doesn't make it broad. And it certainly is not opinion-based. – njsg May 1 '18 at 13:07
0

If there are many files, there is more to search/shuffle around when adding a new file. One way around this is to partition the directory, i.e., create subdirectories for the first character of the name (or thereabouts, git(1) uses the first two hex digits of the object hash which it uses as filename; Fedora package repositories use the first letter in lowercase).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.