AWS s3 CLi grep command with special characters

Question

I want to copy the whole log that are stored in the AWS S3 bucket if the following line is present:

\"Key\" : 951332,\n

I've tried escaping by trying this:

aws s3 ls s3://bucket_name | grep "/\"Key/\" : 951332,/\n" --recursive

but not getting anything back, does anyone know how I can run the grep in this manner?

Answer 1

Mount S3 using s3fs, then do..:

grep -r Key /PATH/TO/MOUNT/POINT/

... then pipe it through grep 951332 and check if that's enough resolution for your case.

It might take a while and incur in AWS DataTransfer cost on you if you run this locally away from AWS, so you ideally want to run this from an EC2 instance in the same VPC.

If you are anyway bold to run locally away from AWS with this approach even with its costs, you might want to redirect stdout and stderr to some docs in order to check them later if necessary, instead of running the costly command line again.

Wrapping up, I might go with:

grep -r Key /PATH/TO/MOUNT/POINT/ | \
        grep 951332 \
        >/LOCAL/PATH/TO/grep_stdout \
        2>/LOCAL/PATH/TO/grep_stderr
#   In /LOCAL/PATH/TO/grep_stdout should be the paths to the docs you
# were searching.

Alternatively:

grep -r Key /PATH/TO/MOUNT/POINT/ | \
        grep 951332 &>/LOCAL/PATH/TO/all_grep_outputs
#   In /LOCAL/PATH/TO/all_grep_outputs should be the paths to the docs you
# were searching.

Answer 2

Cloudgrep is good for this:

./cloudgrep --bucket test-s3-access-logs --query 9RXXKPREHHTFQD77