5

If I give ls -1 I get like this,

file_0001.jpeg
file_0002.jpeg
file_0003.jpeg
file_0004.jpeg
file_0005.jpeg
file_0006.jpeg
file_0007.jpeg
file_0008.jpeg
file_0009.jpeg
file_0010.jpeg
file_0011.jpeg
file_0012.jpeg
file_0013.jpeg
file_0014.jpeg
file_0015.jpeg
file_0016.jpeg
file_0017.jpeg
file_0018.jpeg
file_0019.jpeg
file_0020.jpeg
file_0021.jpeg
...
file_0999.jpeg

Is there a way using awk or other tool to see if some file is missing in this consecutive incremental way.

6
  • 8
    Put list in file, generate second file with file names for all numbers using "for i in seq -w 0 999" etc., use diff to compare files.
    – dirkt
    Oct 13 at 8:02
  • @Sundeep nope, numbers there are based on time, here are incremental numerical, different things, but the most voted answers responds my question Oct 13 at 8:59
  • Ok, how about this one? How to find missing files with sequential names? - use seq -w to get number sequence with leading zeros
    – Sundeep
    Oct 13 at 9:28
  • 3
    Does this answer your question? How to find missing files with sequential names? Please note that posting the same question again is not the accepted way to attract more attention to a lack of unsatisfactory answers.
    – AdminBee
    Oct 14 at 8:37
  • 2
    @AdminBee Please note I have been using SO for almost 10 years so I know the rules, I wasn't aware of the question you suggest at the time I asked mine, I would appreciate if you don't charge me things or say things on my behalf. I noticed when the question brought that is a similar question but remarked that there are better answers on this question. Oct 14 at 13:31
12

If you're using the bash shell, you can use seq or jot to create a "perfect" output reference sample, then compare the output of ls -1 to that reference:

$ diff <(ls -1) <(seq -f 'file_%04g.jpeg' 999)

This will not only show you any missing files, it will also find extraneous files.

1
  • 1
    Btw for a graphical diff tool I invite you to try meld (available on repos, apt install meld)
    – lolesque
    Oct 14 at 8:34
12

While the other answers provide ways to find exactly which files are missing, the original question of seeing if some file is missing can be checked in a more typing-friendly way.

Your list is the output of ls -1, so piping it into wc should give you a number of files matching the name of the last entry.

ls -1 | wc -l

If the count doesn't match the name of the last file, there must be a missing file.

3
  • Try this: mkdir test; cd test; touch file0001.txt file0002.txt file0002a.txt file0002b.txt file0005.txt file0006.txt; ls -1 | wc -l Your solution gives a successful result even though files 3 and 4 are missing. Then try this: rm file*.txt; touch 00README.txt file0001.txt file0002.txt file0003.txt file0004.txt file0005.txt file0006.txt; ls -1 | wc -l Your solution gives a failed result, even though no files are missing.
    – Jim L.
    Oct 14 at 16:19
  • @JimL. The example output in the question doesn't have extra files like that. But sure, yes, this would not work if there are other kinds of files in the directory. Oct 14 at 18:00
  • Granted, but one never knows what's in the "..." part of the OP's input sample, which indeed is the crux of the problem the OP is looking to solve. Even if file0005.jpeg was instead file0005.png ...
    – Jim L.
    Oct 14 at 18:06
11

You could use Awk to filter out the missing ones. On GNU Awk, with support for multi-char FS, you could pipe your result to

awk -F'[_.]' ' $2 != prev+1 { print "file "  prev+1 " missing" }{ prev = $2 }'

Or using perl

perl -F'[_.]' -ane 'if ($F[1] != $prev+1) {printf "file %d missing\n",$prev+1}; $prev=$F[1]'

If there are more gaps anticipated, you could have awk print out the range of file numbers missing. Modifying the above

awk -F'[_.]' ' $2 != prev+1 { print "file(s) "  prev+1 "-" $2-1 " missing" }{ prev = $2 }' file
3
  • 6
    All awks support multi-char FS. It's multi-char RS that requires GNU awk. Your script will behave the same in any awk.
    – Ed Morton
    Oct 13 at 12:48
  • 1
    Wouldn't this miss listing some missing files which have gaps 2 or more from the next files? Like if I have files 1, 2, 5, then 4 won't be reported.
    – justhalf
    Oct 15 at 3:26
  • @justhalf: Its fixed now
    – Inian
    Oct 17 at 10:02
5

You can filter with comm.

For example, while in a directory with such files with a few missing:

$ cd "$(mktemp -d)"
$ touch file_{0001..0999}.jpeg
$ rm file_0388.jpeg file_0795.jpeg

You can filter like this:

$ ls | comm -13 - <(printf 'file_%s.jpeg\n' {0001..0999})
file_0388.jpeg
file_0795.jpeg

comm is a command to compare 2 files, showing which lines are only present in file 1, which are only in file 2, and which are in both. -1 suppresses lines only present in file 1, -2 suppresses lines only present in file 2, and -3 suppresses the lines that are present in both.

-13 is the same as -1 -3, which suppresses all lines except those that are only present in file 2, the generated filenames.

Extraneous files in the directory can be listed by using -23 instead of -13:

$ touch foo.txt bar.txt
$ ls | comm -23 - <(printf 'file_%s.jpeg\n' {0001..0999})
bar.txt
foo.txt

Not using these options, one can see everything differentiated by indentation. Lines only in file 1 don't have indentation, lines only in file 2 have 1 tab indentation, and lines present in both have 2 tabs indentation:

$ ls | comm - <(printf 'file_%s.jpeg\n' {0001..0999})
bar.txt
        file_0001.jpeg
...
        file_0386.jpeg
        file_0387.jpeg
    file_0388.jpeg
        file_0389.jpeg
        file_0390.jpeg
...
        file_0793.jpeg
        file_0794.jpeg
    file_0795.jpeg
        file_0796.jpeg
        file_0797.jpeg
...
        file_0998.jpeg
        file_0999.jpeg
foo.txt
5
  • Can you explain more on what the -13 option do?
    – justhalf
    Oct 15 at 3:34
  • @justhalf I've added an explanation to the answer.
    – JoL
    Oct 15 at 21:14
  • @Quasímodo Totally right. :P I had just taken that printf from that other command without realizing I could simplify.
    – JoL
    Oct 16 at 18:36
  • comm displays 3 columns : lines only in the first file, only in the second and in both. -13 just suppress column 1 and 3. Oct 17 at 10:06
  • @FrédéricLoyer Is that a correction? Seems to be what I said, just reworded.
    – JoL
    Oct 17 at 16:11
4

Use a for loop.

 for i in file_{0000..0999}.jpeg; do if !  test -e "$i" ; then echo "$i doesn't exist"; fi; done

As a script

#! /bin/bash
for i in file_{0000..0999}.jpeg
do 
if !  test -e "$i"
then
echo "$i doesn't exist"
fi
done
4
  • 5
    Please test your answers before posting. The original version of this was full of syntax errors.
    – terdon
    Oct 13 at 9:16
  • 1
    You don't really need a loop, you can just put them all on one ls command line and look for any stderr output. (See my answer) Oct 16 at 2:03
  • @PeterCordes, thanks. I know it spawns a lots of process, personally I liked this one, output of commands as diff. Oct 16 at 10:17
  • 1
    Yeah for script use that's one of the best ones. For interactive use I'd still go with the stderr method because it's faster to type, and you can use globs. Oct 16 at 10:54
3

an easy way to do that is to loop over numbers from seq command

francois@zaphod:/tmp/tata$
 ls
file_0.txt   file_1.txt   file_2.txt   file_3.txt   file_4.txt   file_60.txt  file_70.txt  file_80.txt  file_90.txt
file_10.txt  file_20.txt  file_30.txt  file_40.txt  file_50.txt  file_61.txt  file_71.txt  file_81.txt  file_91.txt
file_11.txt  file_21.txt  file_31.txt  file_41.txt  file_51.txt  file_62.txt  file_72.txt  file_82.txt  file_92.txt
file_12.txt  file_22.txt  file_32.txt  file_42.txt  file_52.txt  file_63.txt  file_73.txt  file_83.txt  file_93.txt
file_13.txt  file_23.txt  file_33.txt  file_43.txt  file_53.txt  file_64.txt  file_74.txt  file_84.txt  file_94.txt
file_14.txt  file_24.txt  file_34.txt  file_44.txt  file_55.txt  file_65.txt  file_75.txt  file_85.txt  file_95.txt
file_15.txt  file_25.txt  file_35.txt  file_45.txt  file_56.txt  file_66.txt  file_76.txt  file_86.txt  file_96.txt
file_16.txt  file_26.txt  file_36.txt  file_46.txt  file_57.txt  file_67.txt  file_77.txt  file_87.txt  file_97.txt
file_17.txt  file_27.txt  file_37.txt  file_47.txt  file_58.txt  file_68.txt  file_78.txt  file_88.txt  file_98.txt
file_18.txt  file_28.txt  file_38.txt  file_48.txt  file_59.txt  file_69.txt  file_79.txt  file_89.txt  file_99.txt
file_19.txt  file_29.txt  file_39.txt  file_49.txt  file_5.txt   file_6.txt   file_7.txt   file_8.txt   file_9.txt
francois@zaphod:/tmp/tata$
 for i in $(seq 0 99) ; do test -f file_$i.txt || echo "file_$i.txt missing" ; done                                    
file_54.txt missing
francois@zaphod:/tmp/tata$

just adapt to your 4 numbers format in your own filenames

1
  • 1
    file_$(printf '%04d' "$i").txt Oct 14 at 15:22
2

(Turns out this question was a duplicate, and what I use is the same as the top answer on How to print the name of missing files in a folder?)


For interactive use in bash, quick and easy to type and remember:
I typically use {01..99} brace expansion to generate the expected series and look for ls errors:

ls file_0{001..999}.jpeg > /dev/null

The redirect hides the stdout listing of existing files, but stderr remains connected to the terminal. I sometimes leave out the >/dev/null and just scroll back in the terminal if there are fewer than ~100 files, because error messages show up first, while ls is still reading through its args before sorting and printing them. That also verifies that I typed it right and my pattern is matching the files I did want (especially if it includes a glob).

For piping / capturing, you could do foo=$(ls ... 2>&1 >/dev/null) to redirect stderr to the pipe, then redirect stdout to /dev/null while leaving stderr going to the shell's pipe. This would be useful for checking for empty/non-empty error output. But in a script, if you want to obtain the names of the missing files, have look at other answers instead of trying to parse it out of ls error messages which may be internationalized etc.

If you want / if necessary, you can use quotes on other parts of the filename, like 'foo bar '{01..22}.jpg. Or even foo\ bar\ {01..22}*.jpg to expand to 'foo bar '01*.jpg / 'foo bar '02*.jpg etc, so it works even if there's some extra unique name for some or all files somewhere other than the sequence numbers.

This works even if your numbering doesn't use leading zeros, e.g. {1..99} instead of {01..99}. If you want to include leading zeros inside the brace range, you can do stuff like {01..09} which works the way you'd hope, instead of factoring them out of the brace expression like I did in the example.


Note that modern Linux systems support very long arg lists, like 128kiB of text. This method does depend on generating a command line with every file on it. That's 100% fine for interactive use: in the rare case where the list is too huge, the shell will tell you about it.

Another answer used a for loop over the list, which would be slower (starting a separate ls for every file), but safe even in the huge, or on a more limited system with much smaller argv limits.

1
  • 1
    This is the fastest to type (except for ls | wc -l, which doesn't report the missing files) and it's very easy to remember. Depending on the directory contents, one may shorten it up with globs: ls *0{001..999}* >/dev/null.
    – Quasímodo
    Oct 16 at 17:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.