Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

$ ls ./dir_with_huge_amount_of_files/errors/

Suppose a directory is full of pictures with unix timestamps, I mean a lot measured in many GBs or even more. Shell-commands like ls will get overflow-style warnings because they are not designed to work with millions (or more) of pictures. How can I manage such huge amount of files? If, for example, I want to find the picture in the middle (according to the timestamp in the name and creation time), is there some file-system that offers a built-in search feature? Which commands would you use? I tried the comfortable ls and find with necessary flags but they were either very slow or generated warnings so I am thinking that either I need better file-system or db or something like that to pre-index the pictures. I basically need one array to which inodes of the photos should be placed in chronological order. How to do that? Later, metadata with unix-timestamps could be added.

[Update]

There is a serious flaw in current answers, people just post sort-of-answers without empirical tests. If they had tested their suggestions, they would probably fail. Hence, I created you a command-line tool by which you can create the sandbox to create the huge amount of files and test your suggestions like with 1e7 amount of files. It can take a long time to generate the files so be patient. If someone knows quicker way to do this, please edit the code. Type python code.py --help to get the help. Have fun!

Usage Example to create a lot of dirred files

$ ls ./data2
ls: ./data2: No such file or directory
$ python testFill.py -n 3 -d 7                                                 
$ tree data2/                                                                  
data2/
|-- 0
|   |-- 1302407302636973
|   |-- 1302407302638022
|   `-- 1302407302638829
|-- 1
|   |-- 1302407302639604
|   |-- 1302407302641652
|   `-- 1302407302642399
|-- 2
|   |-- 1302407302643158
|   |-- 1302407302645223
|   `-- 1302407302646026
|-- 3
|   |-- 1302407302646837
|   |-- 1302407302649110
|   `-- 1302407302649944
|-- 4
|   |-- 1302407302650771
|   |-- 1302407302652921
|   `-- 1302407302653685
|-- 5
|   |-- 1302407302654423
|   |-- 1302407302656352
|   `-- 1302407302656992
`-- 6
    |-- 1302407302657652
    |-- 1302407302659543
    `-- 1302407302660156

7 directories, 21 files

Code testFill.py

# Author: hhh
# License: ISC license

import os, math, time, optparse, sys

def createHugeAmountOfFiles(fileAmount, dirAmount):
   counter = 0
   DENSITY = 1e7
   dir = "./data/"

   do = dir+str(counter)+"/"
   while (os.path.exists(do)):
      counter = counter+1
      do = dir+str(counter)+"/"

   os.mkdir(do)

   for d in range(int(dirAmount)):
      for f in range(int(fileAmount)):
         timeIt = int(time.time()*1e6)
         if (not os.path.exists(do)):
            os.mkdir(do)

         if (timeIt % DENSITY == 0):
            counter = counter+1
            do = dir+str(counter)+"/"

            if (not os.path.exists(do)):
               os.mkdir(do)


         do = dir+str(counter)+"/"
         if(not os.path.exists(do)):
            os.mkdir(do)

         f = open(do+str(timeIt), 'w')
         f.write("Automatically created file to test Huge amount of files.")
         f.close()
      counter = counter +1


def ls(dir):
   for root, dirs, files in os.walk("./data/"+dir):
      print(files)

def rm(dir):
   for root, dirs, files in os.walk("./data/"+dir):
      for f in files:
         os.remove("./data/"+dir+"/"+f)


def parseCli():
   parser = optparse.OptionParser()
   parser.add_option("-f", "--file", dest="filename",
                     help="Location to remove files only in ./Data.", metavar="FILE")
   parser.add_option("-n", "--number", dest="number",
                     help="Number of files to generate", metavar="NUMBER")
   parser.add_option("-r", "--remove", dest="remove",
                     help="Data -dir content to remove", metavar="NUMBER")
   parser.add_option("-d", "--dir", dest="dir",
                     help="Amount of dirs to generate", metavar="NUMBER")
   parser.add_option("-q", "--quiet",
                     action="store_false", dest="verbose", default=True,
                     help="don't print status messages to stdout")

   return parser.parse_args()

def main():
   (options, args) = parseCli()

   if (options.filename):
      ls(options.filename)
   if (options.number and options.dir):
      createHugeAmountOfFiles(options.number, options.dir)
   if (options.remove):
      rm(options.remove)


main()
share|improve this question
2  
@hhh for dataset's on this scale a properly index-ed db is probably the only option –  xenoterracide Mar 25 '11 at 0:43
    
@xenoterracide: but even dbs must implement fast searching with something like arrays, db sounds an overkill. Source for picture-taking thing is here: github.com/fsphil/fswebcam. Perhaps, I could mod it a bit the time it saves the picture so I could append a line with inode-number&unix-time-stamp to file. Now not with the pictures but line, it would be much faster to search for pictures. Or even more easily, each time a picture is saved to a disk, I append a line to a file of its time-stamp. Round-around solution. But won't solve the problem with current pictures so question relevant. –  user2362 Mar 25 '11 at 0:57
    
@hhh what filesystem are you using? or doesn't that matter yet... ext does has some performance enhancing features which may not be on by default. Though even those probably won't deal on the scale you're talking. DB's are optimized for these things and have various indexing solutions to deal with them. for example a btree index is not just a simple array... –  xenoterracide Mar 25 '11 at 1:01
    
@xenoterracide: ext3, not sure either whether it matters. I think the solution I illustrated fixes the problem for future search problem but it does not help at all with current photos, it is very time-consuming to search it. –  user2362 Mar 25 '11 at 1:07
1  
Do you have millions of files in a single directory? If so you might consider splitting them by one or two-levels deep subdirs, based on first chars of the file name, e.g: a/b/abcdef.jpg –  alex Mar 25 '11 at 5:07

2 Answers 2

Try a different shell. I'd recommend trying zsh for instance, and see if it allows more parameters.

If I understand correctly, part of the filename, is a UNIX timestamp. It might be advisable, to divide the files into folders. If the date/time format is a UNIX epoch number, put chunks of fractions of that number, say 10000's, in a separate folder.

If an ISO 8601 timestamp is part of the filename, simply divide by year, month or day.

share|improve this answer
1  
ls and find aren't builtins in either bash or zsh, so it's unclear how switching shells would help in this instance. –  Robin Green Mar 26 '11 at 12:22
    
It's about shell expansion. If the shell can't expand the globbing, this might be the issue. –  polemon Mar 29 '11 at 21:17
    
I made few tests running commands on about 1e6 files, ZSH faces the same problems: "$ cp * Test/ ksh: cp: Argument list too long % rm * zsh: sure you want to delete all the files in /home/user/Downloads [yn]? y zsh: argument list too long: rm % ls * zsh: argument list too long: ls ". Sorry but I cannot see how this is related to the question -1 because it was so easy to test this, create just 1e6 files and run the commands. –  user2362 Apr 9 '11 at 23:57

Would locate (and of course updatedb) be of any help to you?

share|improve this answer
1  
updatedb uses find. –  dave1010 Mar 25 '11 at 13:20
    
@dave1010, sure, but it does so in the background once in a while, so if it is acceptable to the OP to not necessarilly be up to date every minute, but maybe once a day, then schedule updatedb at a quiet hour (or schedule updatedb frequently but at a low priority, which is what it should be at anyway), then using locate is very quick to find what you want. So the key question is how up to date does the DB (or the index for any other such system) need to be. –  asoundmove Mar 25 '11 at 17:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.