6

I want to navigate through all the files in a folder and find out the missing file for a specific date.

Files are partitioned by hour and file name have yyyy-mm-dd-hh formatting.

So between 2017-07-01 and 2017-07-02 there will be 24 files from 2017-07-01-00 through 2017-07-01-23

How can I find missing hourly file if I pass above dates as start and end date?

Appreciate any input!

1
  • For anyone else viewing this page, you are very likely to find a lot of use in the dateutils collection of tools. :) (I'm not affiliated, btw, I just found them to solve my own problem.)
    – Wildcard
    Jul 9 '19 at 22:56
9
# presuming that the files are e. g. template-2017-07-01-16:

# To test a given date
for file in template-2017-07-01-{00..23}; do
  if ! [[ -f "$file" ]]; then
    echo "$file is missing"
  fi
done

# To test a given year
year=2017
for month in seq -w 1 12; do
    dim=$( cal $( date -d "$year-$month-01" "+%m %Y" | awk 'NF { days=$NF} END {print days}' )
    for day in $(seq -w 1 $dim); do
        for file in template-${year}-${month}-${day}-{00..23}; do
           if ! [[ -f "$file" ]]; then
             echo "$file is missing"
           fi
        done
    done
done
5
  • Why did you put the ! outside of the brackets? Is this because double brackets? I normally would write that as if [[ ! -f "$file" ]]; then so I'm wondering if I'm wrong for doing that or what the difference is? It appears to work either way for me.
    – jesse_b
    Aug 10 '17 at 16:42
  • 3
    Either will work; it's a question of if your thinking is more in line with "if ( not foo )" or "if not (foo)"; they're logically indistinguishable in this case. They're "If not (file exists)" and "If (file does not exist)".
    – DopeGhoti
    Aug 10 '17 at 16:49
  • I am getting error here: cal $( date -d "$year-$month-01" "+%m %Y" | awk 'NF { days=NF} END {print days}' I can run this cal $(date +"%m %Y") | awk 'NF {DAYS = $NF}; END {print DAYS}' any specific reason why the former is not working ?
    – Yu Ni
    Aug 10 '17 at 19:10
  • Are you running GNU date or BSD date?
    – DopeGhoti
    Aug 10 '17 at 19:47
  • The way you found the last day of each month is seriously slick! It will even effortlessly handle leap years.
    – Joe
    Aug 12 '17 at 5:53
6

On a GNU system:

#! /bin/bash -
ret=0
start=${1?} end=${2?}
t1=$(date -d "$start" +%s) t2=$(date -d "$end" +%s)

for ((t = t1; t < t2; t += 60*60)); do
  printf -v file '%(%F-%H)T' "$t"
  if [ ! -e "$file" ]; then
    printf >&2 '"%s" not found\n' "$file"
    ret=1
  fi
done
exit "$ret"

Note that on the day of the switch to winter time (in timezones that implement daylight saving), you may get an error message twice if a file is missing for the hour of the switch. Fix $TZ to UTC0 if you want 24 hours per day for every day (for instance, if whatever creates those files uses UTC time instead of local time).

5
  • -e will also match non-regular files (e. g. sockets, directories, et cetera).
    – DopeGhoti
    Aug 10 '17 at 17:10
  • 1
    @DopeGhoti, true, though the OP didn't specify the type of file he was looking for, nor what should be done if a file was found but with the wrong type (assuming a specific type is expected). Note that [[ -f file ]] also returns true for files of type symlink provided the symlink resolves to a regular file. Also note that [ -e file ] would return false if file was a symlink that can't be resolved by the bash process. Aug 10 '17 at 17:11
  • 1
    What does start=${1?} do? If it was ${1:?} it would throw an error for a missing parameter, but I don't see what it means without the colon. And, since I don't see a -e anywhere, it won't stop the script. Also, it took me awhile to figure out the magic of that printf statement. That's a lot of work getting done in a tiny command!
    – Joe
    Aug 12 '17 at 6:19
  • 1
    @Joe, ${1?} exits the shell (regardless of set -e) with an error if the first argument is not supplied. ${1:?} would exit the shell if it was not supplied or if it was supplied but as an empty string. That's standard Bourne and POSIX syntax. Aug 12 '17 at 6:34
  • 1
    @Joe I had the same question and found this: Parameter Expansion. From there - "the <colon> in the format shall result in a test for a parameter that is unset or null; omission of the <colon> shall result in a test for a parameter that is only unset. If parameter is '#' and the colon is omitted, the application shall ensure that word is specified (this is necessary to avoid ambiguity with the string length expansion)."
    – MiniMax
    Aug 12 '17 at 9:01
4

What about command like below:

 grep -Fvf <(find * -type f \( -name "2017-07-02-00" $(printf " -o -name %s" 2017-07-02-{01..23}) \)) \
           <(printf "%s\n" 2017-07-02-{00..23})
ls
2017-07-02-01  2017-07-02-06  2017-07-02-08  2017-07-02-14  2017-07-02-19
2017-07-02-04  2017-07-02-07  2017-07-02-11  2017-07-02-15  2017-07-02-22

The output after command ran:

2017-07-02-00
2017-07-02-02
2017-07-02-03
2017-07-02-05
2017-07-02-09
2017-07-02-10
2017-07-02-12
2017-07-02-13
2017-07-02-16
2017-07-02-17
2017-07-02-18
2017-07-02-20
2017-07-02-21
2017-07-02-23

Above we are generating all possibilities of 24 files using printf and pass it to find its -name parameter which printf also helping her, then with grep command we are printing those files are exist in our pattern but find didn't find them.

0
1

Usage: ./diff_date.sh 2017-08-30-00 2017-09-02-00

#!/bin/bash

# This processing is needed, because `date` require 2017-08-30 00 format,
# not 2017-08-30-00. So, last dash is replacing by space in here.
start=$(sed 's/-/ /3' <<< "$1")
end=$(sed 's/-/ /3' <<< "$2")

while [[ "$start" != "$end" ]]; do
    # Returns dash back to its place and checks - does this file exist. 
    if [ ! -f "${start/ /-}" ]; then 
        echo "${start/ /-}"
    fi  
    # Performance of this code can be improved, by calling `date` only when
    # day is changing, not the every hour.
    start=$(date -d "${start} + 1 hour" "+%F %H")
done

Testing:

# make files
$ touch 2017-08-{30..31}-{03..23}; touch 2017-09-{01..02}-{03..23}
$
$ ./diff_date.sh 2017-08-30-00 2017-09-02-00
##### Output - missing files. #####
2017-08-30-00
2017-08-30-01
2017-08-30-02
2017-08-31-00
2017-08-31-01
2017-08-31-02
2017-09-01-00
2017-09-01-01
2017-09-01-02
1
  • 1
    +1 - I would just feel better if [[ "$start" != "$end" ]] was [[ "$start" <= "$end" ]]. I'm not sure about the = - to catch the last possible missing value. The main point is to avoid an infinite loop if start isn't less than end.
    – Joe
    Aug 12 '17 at 6:44
0

Why not use egrep? you can then regex it the way you want.

 egrep (2017-07-0[1-2]-\d\d$) *file name here*| tail     

regex might be a little off - sorry.

1
  • from egrep --help: Invocation as 'egrep' is deprecated; use 'grep -E' instead.
    – phuclv
    Aug 11 '17 at 2:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.