Copy files using scp

Question

I am copying 6 files through scp command with the use of wildcards.

All the files are appended with the date(yyyymmddhhmmss) when the files were created. Example :

File1.csv.rpl.20210702221033
File2.csv.rpl.20210702221523
...
File6.csv.rpl.20210703003011

All these files are created as part of an overnight batch run so what happens is one file gets generated after 12 am, hence the timestamp appended is of new day.

When I am trying to copy these files using wildcard(scp *20210703*) all the files for 02nd July are getting copied thus copying yesterdays one file as well.

What I am basically looking for is if I want to copy files for date 20210701, the 5 files that are getting generated on 1st july should be copied along with the one file that's getting generated after 12am. Is it possible ?

Answer 1

If you're have a version of date that allows for date arithmetic it seems to be fairly straightforward. The assumption here is that the overlapped file has a timestamp during 00:00 (midnight) - 01:00 (am), so we can pick it out with a 00 hour component. Is this assumption isn't valid please update your question with more information relating to the time or other conditions of the overlapped file name

#!/bin/bash
shopt -s nullglob

today=$(date +%Y%m%d)
yesterday=$(date --date "$today -1 day" +%Y%m%d)

echo "Will copy" *."$yesterday"* *."$today"00* >&2
scp -p *."$yesterday"* *."$today"00* remoteHost:...

I have set nullglob so that unmatched wildcards disappear when expanded rather than being left as a string literal containing an asterisk.