math problem or variable value problem im not sure

Question

I am having an issue comparing epoch times. I'm really not sure what I'm doing wrong. I have been tasked with identifying and actioning files beyond and during certain date ranges.

In this case, it's a cleanup script. It takes a few parameters to do the search and then checks the dates. My dates are like this:

earliest date we can delete a file -> date -d "-30days" +"%s"
earliest date we can compress a file -> date -d "-14days" +"%s"
get the file last modified time: stat -c "%y" "/path/to/file"
get the epoch from that last modified time: date -d "<from the obtained time from stat>" +"%s"

The math goes like this:

if (( FILE_TIMESTAMP >= EARLIEST_DELETE_DATE ))
then
    delete file;
elif (( FILE_TIMESTAMP >= EARLIEST_COMPRESS_DATE ))
then
    compress file;
else
    echo "file skipped as it wasnt in teh date ranges";
fi

(edit to the above code: I removed the check for <= EARLIEST_DELETE_DATE here)

The first if statement works as expected. The second works halfway - it does identify that the timestamp is greater than the compress date but the check for less than the delete date appears to be returning false as that section of code never gets run.

Am I doing the math wrong or do I have the variables wrong?

Also, I know my posts are crap. I'll delete my account after this so as to lessen the strain on the community.

Some updates:

Looking at my debug log output I see this:

[Time: ][Thread: 883837][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:99] [Method: manageFileLifecycle.sh] - FILE_DATESTAMP -> 2023-05-07 03:20:29.000000000 -0400 [Time: ][Thread: 883837][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:100] [Method: manageFileLifecycle.sh] - FILE_TIMESTAMP -> 1683444029

The file shown is older than 6 months but was not triggered by the >= delete clause. not sure what I'm doing wrong

For @markp-fuso:

declare -- FILE_TIMESTAMP="1688178017"
declare -i EARLIEST_DELETE_DATE="1697037095"
declare -i EARLIEST_COMPRESS_DATE="1698419495"
FILE_TO_PROCESS -> /opt/cws/log/S93206319.log, FILE_DATESTAMP -> 2023-06-30 22:20:17.000000000 -0400 FILE_TIMESTAMP -> 1688178017, EARLIEST_DELETE_DATE -> 1697037095, EARLIEST_COMPRESS_DATE -> 1698419495

In the log output I see this file being skipped, even though it shouldn't have been.

Also, I must apologize - I cannot share the whole script as it is company property =(

More information:

Specific file pulled that ISN'T working:

declare -- FILE_TIMESTAMP="1679107439"
declare -i EARLIEST_DELETE_DATE="1697040305"
declare -i EARLIEST_COMPRESS_DATE="1698422705"
FILE_TO_PROCESS -> /opt/cws/log/S93382366.log, FILE_DATESTAMP -> 2023-03-17 22:43:59.000000000 -0400 FILE_TIMESTAMP -> 1679107439, EARLIEST_DELETE_DATE -> 1697040305, EARLIEST_COMPRESS_DATE -> 1698422705
file skipped as it wasnt in teh date ranges

and

[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:93] [Method: manageFileLifecycle.sh] - processRequestedFile -> enter
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:94] [Method: manageFileLifecycle.sh] - Provided arguments: /opt/cws/log/S93382366.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:95] [Method: manageFileLifecycle.sh] - ERROR_COUNT -> 0
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:96] [Method: manageFileLifecycle.sh] - FILE_TO_PROCESS -> /opt/cws/log/S93382366.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:97] [Method: manageFileLifecycle.sh] - FILE_CORE_NAME -> S93382366
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:98] [Method: manageFileLifecycle.sh] - FILE_CORE_EXT -> log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:99] [Method: manageFileLifecycle.sh] - FILE_DATESTAMP -> 2023-03-17 22:43:59.000000000 -0400
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:100] [Method: manageFileLifecycle.sh] - FILE_TIMESTAMP -> 1679107439
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:107] [Method: manageFileLifecycle.sh] - FILE_CORE_EXT and FILE_CORE_NAME do not match - executing with a file extension
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:139] [Method: manageFileLifecycle.sh] - File has extension log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:298] [Method: manageFileLifecycle.sh] - RETURN_CODE -> 0
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:299] [Method: manageFileLifecycle.sh] - processRequestedFile -> exit

A file that appears to have at least hit the delete clause:

Output from the typeset:

declare -- FILE_TIMESTAMP="1703308529"
declare -i EARLIEST_DELETE_DATE="1697040305"
declare -i EARLIEST_COMPRESS_DATE="1698422705"
FILE_TO_PROCESS -> /opt/cws/log/S20504215.log, FILE_DATESTAMP -> 2023-12-23 00:15:29.000000000 -0500 FILE_TIMESTAMP -> 1703308529, EARLIEST_DELETE_DATE -> 1697040305, EARLIEST_COMPRESS_DATE -> 1698422705

Output from the debug log:

[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:93] [Method: manageFileLifecycle.sh] - processRequestedFile -> enter
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:94] [Method: manageFileLifecycle.sh] - Provided arguments: /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:95] [Method: manageFileLifecycle.sh] - ERROR_COUNT -> 0
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:96] [Method: manageFileLifecycle.sh] - FILE_TO_PROCESS -> /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:97] [Method: manageFileLifecycle.sh] - FILE_CORE_NAME -> S20504215
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:98] [Method: manageFileLifecycle.sh] - FILE_CORE_EXT -> log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:99] [Method: manageFileLifecycle.sh] - FILE_DATESTAMP -> 2023-12-23 00:15:29.000000000 -0500
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:100] [Method: manageFileLifecycle.sh] - FILE_TIMESTAMP -> 1703308529
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:107] [Method: manageFileLifecycle.sh] - FILE_CORE_EXT and FILE_CORE_NAME do not match - executing with a file extension
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:139] [Method: manageFileLifecycle.sh] - File has extension log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:143] [Method: manageFileLifecycle.sh] - File timestamp is greater than or equal to the delete date. File -> /opt/cws/log/S20504215.log, Timestamp -> 1703308529, EARLIEST_DELETE_DATE -> 1697035331
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:148] [Method: manageFileLifecycle.sh] - Deleting file /opt/cws/log/S20504215.log as it falls after the earliest delete date range.
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:149] [Method: manageFileLifecycle.sh] - EXEC: deleteRequestedFile /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:462] [Method: manageFileLifecycle.sh] - deleteRequestedFile -> enter
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:463] [Method: manageFileLifecycle.sh] - Provided arguments: /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:464] [Method: manageFileLifecycle.sh] - REMOVABLE_FILE -> /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:465] [Method: manageFileLifecycle.sh] - REMOVABLE_FILE_NAME -> S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:466] [Method: manageFileLifecycle.sh] - REMOVABLE_FILE_PATH -> /opt/cws/log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:470] [Method: manageFileLifecycle.sh] - Checking if /opt/cws/log exists...
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:516] [Method: manageFileLifecycle.sh] - Returning to previous directory /opt/cws/log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:520] [Method: manageFileLifecycle.sh] - Previous directory: /opt/cws/log, New working directory: /opt/cws/log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:530] [Method: manageFileLifecycle.sh] - RETURN_CODE -> 1
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:531] [Method: manageFileLifecycle.sh] - deleteRequestedFile -> exit

The goal is to delete any file older than 30 days from the current date, compress anything older than 14 days from the current date but not older than 30 days from the current date (these entries would fall under the delete) and anything younger than 14 days is left alone.

If file_timestamp >= earliest_delete_date is false, which must be the case for the shell to move from the if condition to the else conditions, then by definition file_timestamp <= earliest_delete_date (well, technically it's strictly less than, but anyway) so that check is useless and is always true when it is recahed. But it seems to me your conditions are inverted - shouldn't the timestamp of a file that can be deleted be less than the earliest_delete_date? — muru, Commented Nov 10, 2023 at 13:40
"I know my posts are crap. I'll delete my account after this so as to lessen the strain on the community" - that would be a shame. The community is made of people, and so you are implicitly part of that community. Ask away — Chris Davies, Commented Nov 10, 2023 at 15:21
Regarding I must apologize - I cannot share the whole script as it is company property - you're not expected to post your real script, you're expected to create a minimal, complete script that just demonstrates the problem you're asking for help with along with concise, testable sample input and expected output that can reproduce that problem when the script you provide is run on it. — Ed Morton, Commented Nov 10, 2023 at 16:35
Regarding I know my posts are crap. I'll delete my account after this so as to lessen the strain on the community. - if you do that then any help others with similar problems may get from reading this in future and any points awarded to anyone who answers you will be wiped out so you won't be helping the community in any way by deleting your account. It'd be much better for you and everyone else if you just learned how to ask a good question (same rules here as anywhere else in life), see How to Ask and look at questions that have been upvoted and answered for examples. — Ed Morton, Commented Nov 10, 2023 at 16:48
Looks like you got answers posted now but make sure to consider what the timestamp should be for files you compress - should they end up having todays timestamp or the timestamp of the original file. — Ed Morton, Commented Nov 10, 2023 at 16:55

Stéphane Chazelas · Accepted Answer · 2023-11-10 17:05:06Z

You have your logic reversed. You're treating those timestamps as if they were ages. Should be:

if (( FILE_TIMESTAMP < EARLIEST_DELETE_DATE ))
then
    delete file
elif (( FILE_TIMESTAMP < EARLIEST_COMPRESS_DATE ))
then
    compress file
else
    echo "file skipped as it wasn't in the date ranges"
fi

You could also do it portably with find:

find -- "$file" -prune -mtime +29 -exec rm -f -- {} + -o \
  -mtime +13 -exec gzip -- {} + -o \
  -exec echo Skipping {} +

Note that despite the --, that fails for values of $file that start with - (or are (, ), !). gzip also has a problem with the file called - (which it treats as meaning standard input instead; same for GNU stat).

-mtime +29 is true for files whose age rounded down to an integer number of days is strictly greater than 29, so that's files that are 30 days old or older.

With the find of BSDs, both issues/confusions can be addressed by doing:

find -f "$file" -prune -mtime +30d -exec rm -f -- {} + -o \
  -mtime +14d -exec gzip -- {} + -o \
  -exec echo Skipping {} +

(note the d suffix).

While with recent versions of GNU find, one can do:

find  -files0-from <(printf '%s\0' "$file") -prune \
  ! -newermt '30 days ago' -exec rm -f -- {} + -o \
  ! -newermt '14 days ago' -exec gzip -- {} + -o \
  -printf 'Skipping %p\n'

Do not user -delete in place of -exec rm -f {} + as -delete implies -depth and do not use -exec ... ';' in place of -exec ... + in the first 2 -execs in those commands as then the success/failure of the command would be used in the find logic.

All those commands can take more than one file to process (with BSD find, use -f "$file1" -f "$file2"...).

you are correct, i changed and it works perfectly. yes I'm truly horrible at math — Kevin Huntly, Commented Nov 10, 2023 at 16:53

markp-fuso · Accepted Answer · 2023-11-10 20:48:21Z

At the end of the question we have the following rules:

delete any file older than 30 days from the current date
compress any file older than 14 days from the current date but not older than 30 days from the current date (these entries would fall under the delete)
any file younger than 14 days is left alone

From the latest update we're given two sets of epoch values:

filename: /opt/cws/log/S93382366.log

        FILE_TIMESTAMP="1679107439"      # Fri Mar 17 21:43:59 CDT 2023
  EARLIEST_DELETE_DATE="1697040305"      # Wed Oct 11 11:05:05 CDT 2023
EARLIEST_COMPRESS_DATE="1698422705"      # Fri Oct 27 11:05:05 CDT 2023

    code result: ISN'T working (assuming this means code generates "file skipped as it wasnt in the date ranges")

expected result: delete file (more than 30 days old; NOTE: compression should have occurred a long time ago)

-----------

filename: /opt/cws/log/S20504215.log

        FILE_TIMESTAMP="1703308529"      # Fri Dec 22 23:15:29 CST 2023
  EARLIEST_DELETE_DATE="1697040305"      # Wed Oct 11 11:05:05 CDT 2023
EARLIEST_COMPRESS_DATE="1698422705"      # Fri Oct 27 11:05:05 CDT 2023

    code result: delete file

expected result: file skipped as it wasn't in the date ranges (less than 14 days old; actually, file has timestamp that is 42 days in the future when today = Nov 10 2023)

Primary issue with the code:

an 'older' epoch date is actually going to be less than a 'newer' epoch date so ...
the code needs to be testing for epochs that are less than a desired threshold

Correcting the current code to implement the 3 desired rules:

if (( FILE_TIMESTAMP < EARLIEST_DELETE_DATE ))
then
    echo "delete file"
elif (( FILE_TIMESTAMP < EARLIEST_COMPRESS_DATE ))
then
    echo "compress file"
else
    echo "file skipped as it wasn't in the date ranges";
fi

If we apply this to our 2 sample files/epoch-tuples we get:

filename: /opt/cws/log/S93382366.log

        FILE_TIMESTAMP="1679107439"      # Fri Mar 17 21:43:59 CDT 2023
  EARLIEST_DELETE_DATE="1697040305"      # Wed Oct 11 11:05:05 CDT 2023
EARLIEST_COMPRESS_DATE="1698422705"      # Fri Oct 27 11:05:05 CDT 2023

code result: delete file

-----------

filename: /opt/cws/log/S20504215.log

        FILE_TIMESTAMP="1703308529"      # Fri Dec 22 23:15:29 CST 2023
  EARLIEST_DELETE_DATE="1697040305"      # Wed Oct 11 11:05:05 CDT 2023
EARLIEST_COMPRESS_DATE="1698422705"      # Fri Oct 27 11:05:05 CDT 2023

code result: file skipped as it wasn't in the date ranges

you are correct, i changed and it works perfectly. yes I'm truly horrible at math — Kevin Huntly, Commented Nov 10, 2023 at 16:53

Stack Exchange Network

math problem or variable value problem im not sure

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
bash
shell-script
math
.

Hot Network Questions

math problem or variable value problem im not sure

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged bashshell-scriptmath.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
bash
shell-script
math
.