1

I have a bash script generating csv files in the following format:

056_log_202312290735.csv
056_log_202312290736.csv
067_dat_202312290737.csv
067_dat_202312290838.csv
056_log_202312290951.csv
067_dat_202312290952.csv
056_log_202312290953.csv
...
056_log_YYYYmmddHHMM.csv
067_dat_YYYYmmddHHMM.csv

where YYYYmmddHHMM is the timestamp of the data contained in the csv file itself, not the creation time of the csv file.

So I would like write a bash script to compare current system time (in the format same as that of the csv file, i.e YYYYmmddHHMM) with timestamp on the filename,

If the difference between the two is greater or equal to 120 minutes, files must be moved to old_data directory.

If the difference between the two is less than 120 minutes, files must be moved to current_data directory

Using current system time as 202312291048, the files must be moved as follows:

~/old_data/
056_log_202312290735.csv
056_log_202312290736.csv
067_dat_202312290737.csv
067_dat_202312290838.csv
~/current_data/
056_log_202312290951.csv
067_dat_202312290952.csv
056_log_202312290953.csv

So far, I know I can get current time in the format I want with:

CUR_TIME="`date +%Y%m%d%H%M`";

and get timestamps from csv files with:

ls 056*.csv | cut -d'_' -f 3 | cut -c -12; #get timestamps from 056 files
ls 067*.csv | cut -d'_' -f 3 | cut -c -12; #get timestamps from 067 files

From there onwards, I don't know how to move forward.. please help.

this is what i could think of:

#!/bin/bash

CUR_TIME=$(date +%Y%m%d%H%M);

for csvfile in *.csv
do
    TIME_DIFF=0
    TIMESTAMP= $(echo $csvfile | cut -d'_' -f 3 | cut -c -12)
    TIME_DIFF= $CUR_TIME-$TIMESTAMP
    
    if $TIME_DIFF >= 120
    then
    mv -f $csvfile ~/old_data/
    else
    mv -f $csvfile ~/current_data/
    fi
    
done
4
  • Firstly, don't parse the output of ls. Use printf %s\n' 056*.csv`. Secondly, what have you tried beyond that? Do you not understand how to use conditional statements or move data from directory to another? Commented Dec 29, 2023 at 12:36
  • Hi Nasir.. I can manually move data from directory to another and yes, I don't know how to use conditional statements... fairly new to bash scripting
    – dante_t
    Commented Dec 29, 2023 at 13:03
  • Do you want the current time to change while the files are being processed? I mean, what if you have millions of files and it takes a few minutes to go through them, should we always compare to the time when the script started or should we compare to the time at the moment when each file is being processed?
    – terdon
    Commented Dec 29, 2023 at 13:54
  • What does if $TIME_DIFF >= 120 do? Or TIME_DIFF= $CUR_TIME-$TIMESTAMP? Did you try that script?
    – ilkkachu
    Commented Dec 29, 2023 at 14:02

1 Answer 1

1

You cannot simply subtract datestamps like that since, for example, 202312291607 (which is 2023/12/29 16:07) minus 60 is 202312291547 which is actually 2023/12/29 15:47 so not 60 minutes ago but instead 20 minutes ago. Remember that we measure time using a base 60 system, so we cannot then do simple base 10 calculations as you propose. The common solution is to convert your times to seconds since the epoch, then compare them to get the difference in seconds, and then convert that to minutes. For example:

#!/bin/bash

## Get the current time in seconds since the epoch
curr_time=$(date +%s)

for csvfile in  *.csv; do
  ## Extract the timestamp from the csv file name. 
  csv_date_string=$(basename "${csvfile##*_}" .csv |
                      sed -E 's|(....)(..)(..)(..)(..)|\1/\2/\3 \4:\5|')
  ## Convert the csv datestamp to seconds since the epoch 
  csv_time=$( date -d "$csv_date_string" +%s)

  ## Compare to the current time and, if more than or equal to 120
  ## echo the mv command, if less, echo that we do nothing.
  if [[ $(( (curr_time - csv_time) / 60)) -ge 120 ]]; then
    echo mv "$csvfile" old_data
  else
    echo "Not moving $csvfile"
  fi
  
done

Run the script above and, if it looks like it works as you want, remove the echo to actually run the mv command and move the files.

The trick here is using basic string manipulation to extract the timestamp from the file name. The syntax ${var##pattern} will remove the longest match of pattern from the beginning of a string. Here, the pattern is *_, so everything until an _. Her it is in action:

$ csvfile=056_log_202312290736.csv
$ echo ${csvfile##*_}
202312290736.csv

So ${csvfile##*_} removes everything until the last _, leaving us with the datestamp plus .csv. The basename command is designed to remove paths from a file name and leave only the file name, but it also has a useful trick of removing the provided extension, so that's why I used basename with .csv to get the actual timestamp:

$ basename "${csvfile##*_}" .csv
202312290736

The rest is relatively straightforward. A simple if to check if the time difference is more than 120 minutes. Note that since I am working with times in seconds, I had to divide by 60 t get minutes. Alternatively, I could have converted 120 minutes to 7200 seconds (120 x 60):

if [[ $(( curr_time - csv_time )) -ge 7200 ]]; then

Important: If you have very many files, and the processing can take minutes or even hours, note that each file will still be compared to the time when the script launched. This means that things that could be older than 120 minutes by the time the script gets to them, but were not so when the script started, will not be moved. If you don't want this, then move the curr_time=$(date +%s) line inside the for loop so it is reset for every file.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .