1

I have a folder which subfolder naming convention is like yyyymm for e.g. 200801, 200803 etc. I want to list down folder after 201412 via command line means to upload to S3. Is there a way to do it?

  • 1
    don't we fave files in the same location and don't we have any folderName with string in the same location? – msp9011 Mar 26 at 6:07
  • given the bash tag, I won't answer with this, but consider zsh. If you can safely assume the date/directory name format (e.g. no 201413 directory), then zsh -c 'print -l <201413->(/)' would get them. – Jeff Schaller Mar 27 at 21:59
  • Welcome to U/L! 6 answers and no upvotes to the question… have a +1! – Sparhawk Mar 27 at 22:37
1

list directory content, pipe output to awk, print only those which values are greater or equal than the date you want, pipe to xargs s3.

ls | awk '$1 >= 201412 {print $1}' | xargs -n1 YOUR_S3_COPY_COMMAND -options

If it is a lot of of directories and files, you can parallelize xargs with -P number of concurrent processes.

ls | awk '$1 >= 201412 {print $1}' | xargs -n1 -P20 YOUR_S3_COPY_COMMAND -options

  • it says Unknown options: 201501,201502,201503,201504,201505,201506,201507,201508,201509,201510,201511,201512,201601,201602,201603,201604,201605,201606,201607,201608,201609,201610,201611,201612,201701,201702,201703,201704,201705,201706,201707,201708,201709,201710,201711,201712,201801,201802,201803,201804,201805,201806,201807,201808,201809,201810,201811,201812,201901,201902,201903 – Moses Liao GZ Mar 28 at 8:50
  • show me the command you're running. – Jakub Jindra Mar 28 at 8:52
  • ok i got it. it needs the -n1 in the xargs command I was running ls | awk -v snumber=201412 '$1>snumber {print $1}' | xargs aws s3 sync {$1} s3://s3-static-zaobao/s3fs-public/images/ --acl public-read --dryrun – Moses Liao GZ Mar 28 at 8:55
  • You're right. I'll edit the answer. – Jakub Jindra Mar 28 at 8:57
1

Your current requirement is to list subdirectory names that are after 201412, i.e. 201501 or later.

These directories would match the pattern 201[5-9]*.
To also handle names after 2019 (until 2099), use 201[5-9]* 20[2-9]*.

To list them:

shopt -s nullglob # to allow non-matching globs to expand to nothing
printf '%s\n' somedir/201[5-9]*/ somedir/20[2-9]*/

where somedir is the directory where your subdirectories are kept.

To do something with each of these:

shopt -s nullglob
for dir in somedir/201[5-9]*/ somedir/20[2-9]*/; do
   aws s3 cp "$dir" ...
done

Note: I've never used S3, so I would not know the correct command to use for uploading a directory of files.

0

I am not sure you can do it directly so you need to get the list of directories and then filter the output. Command like this can help:

ls|awk -v snumber=201412 '$1>snumber {print}'
0

Try this,

 find . -type d  -regex '^./[0-9]+' -exec sh -c 'test "{}" ">" "./201412"' \; -print

As I have queried in comments. The current directory may have files in and non-numeric directory names

  • type -d will list only the directories
  • -regex '^./[0-9]+' will list only the names with numeric
  • test is used to compare the result

Here you can add maxdepth to restrict the search to current directory, but our regex itself will do the same.

  • There's no real reason to use sh -c with -exec here, is there? – Kusalananda Mar 27 at 19:12
0

i have just listing you replace ls command with as per your requirement

ls -ltr | awk '$1 ~ /^d/{print $0}'|sed -n '/[0-9]\{6\}/p'| awk '$NF >= "201412" {print "ls -ltr "$NF}'| sh
0

I would use a variation of a now-deleted answer to build up an array of desired files in bash:

$ mkdir 201412 201501 201705 202003

$ files=()
$ for d in */
do
  (( d > 201412)) && files+=("$d")
done
$ declare -p files
declare -a files='([0]="201501" [1]="201705" [2]="202003")'

You could then safely loop over that array (or expand it all out) to upload it, such as:

$ aws upload command "${files[@]}"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.