I am looking for a way to determine file types in a folder with thousands of files. File names do not reveal much and have no extension, but are different types. Specifically, I am trying to determine if a file is a sqlite database.

When using the file command, it determines the type of 2-3 files per second. This seems like a good way to address the problem, except it is too slow.

Then I tried opening each file with sqlite3 and checking to see if I get an error. That way, I can check 4-5 files per second. Much better, but I think that there might be a better way to do this.

  • As a timing reference, file run on 2115 files with no special options takes 0m0.117s user 0m0.078s sys 0m0.044s for me
    – Glenn Plas
    Nov 1, 2012 at 0:14

3 Answers 3


2-3 files per second tested with file seems very slow to me. file actually performs a number of different tests to try and determine the file type. Since you are looking for one particular type of file (sqlite), and you don't care about identifying all the others, you can experiment on a known sqlite file to determine which test actually identifies it. You can then exclude the others using the -e flag, and run against your full file set. See the man page:

 -e, --exclude testname
         Exclude the test named in testname from the list of tests made to
         determine the file type. Valid test names are:

            EMX application type (only on EMX).
            Various types of text files (this test will try to guess the
            text encoding, irrespective of the setting of the ‘encoding’
            Different text encodings for soft magic tests.
            Looks for known tokens inside text files.
            Prints details of Compound Document Files.
            Checks for, and looks inside, compressed files.
            Prints ELF file details.
            Consults magic files.
            Examines tar files.

Edit: I tried some tests myself. Summary:

  1. Applying my advice with the right flags can speed up file by about 15%, for tests to determine sqlite. Which is something, but not the huge improvement I expected.
  2. Your file tests are really slow. I did 500 on a standard machine in the time you did 2-3. Are you on slow hardware, or checking enormous files, running an ancient version of file, or...?
  3. You must keep the 'soft' test to successfully identify a file as sqlite.

For a 16MB sqlite DB file, I did:

for  i in {1..1000}
    file sqllite_file.db | tail > out

Timing on the command line:

~/tmp$ time ./test_file_times.sh; cat out

real    0m2.424s
user    0m0.040s
sys 0m0.288s
sqllite_file.db: SQLite 3.x database

Trying the different test excludes, and assuming the determination is made based on a single test, it is the 'soft' (i.e. magic file lookup) test which identifies the file. Accordingly, I modified the file command to exclude all the other tests:

file -e apptype -e ascii -e encoding -e tokens -e cdf -e compress -e elf -e tar sqllite_file.db | tail > out

Running this 1000 times:

~/tmp$ time ./test_file_times.sh; cat out

real    0m2.119s
user    0m0.060s
sys         0m0.280s
sqllite_file.db: SQLite 3.x database
  • Thanks for the elaborate answer. I also found out that 'soft' test is required. Turning off the other tests (on 10000 iterations) gave roughly the same result.
    – dmars
    Sep 21, 2012 at 23:08
  • After your comment that the execution time is too slow I started questioning the procedure that runs it and I found out that sqlite=$(echo $filetype | grep -c SQLite) requires at least the same time as file command. And I have some other commands additionally slowing down the procedure. Interestingly though, running sqlite3 $filename ".schema" was still two times faster than file command.
    – dmars
    Sep 21, 2012 at 23:19

If you look at: http://www.sqlite.org/fileformat.html, the SQLite format starts with the string "SQLite format 3\000". It seems to me that you could check head -c 16 of the file to check the format. I'd expect this to be faster then using more general purpose tools.


If you look at file's magic description for sqlite files, it looks for either SQLite format 3 or ** This file contains an SQLite at the beginning of the file.

So you could either create a magic file with just those checks (and disable the builtin tests as in @ire_and_curses solution) or do the check manually:

case $(head -c 31 < "$file") in
  ("** This file contains an SQLite"*) echo sqlite 2;;
  ("SQLite format 3"*) echo sqlite 3;;

Not very efficient as it runs a head per file. With a little effort you, you could probably do it in perl, to read the first 31 bytes of several files in one perl invocation.

  • Thanks. Turned out that trying to open with sqlite3 was still the faster way. Regarding the performance, the problem was mostly with other commands in the procedure that I haven't considered.
    – dmars
    Sep 21, 2012 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.