2

I want to copy the whole log that are stored in the AWS S3 bucket if the following line is present:

\"Key\" : 951332,\n

I've tried escaping by trying this:

aws s3 ls s3://bucket_name | grep "/\"Key/\" : 951332,/\n" --recursive

but not getting anything back, does anyone know how I can run the grep in this manner?

2
  • Your grep is getting the output of aws s3 ls as input. The --recursive option makes no sense as you don't actually want to run that locally. You would have to look inside the files on the AWS side of things. I don't know how to do that as I've never used AWS.
    – Kusalananda
    Commented Sep 2, 2019 at 12:08
  • 2
    About that line. Are you wanting to match that literally, including the backslashes?
    – Kusalananda
    Commented Sep 2, 2019 at 12:13

2 Answers 2

0

Mount S3 using s3fs, then do..:

grep -r Key /PATH/TO/MOUNT/POINT/

... then pipe it through grep 951332 and check if that's enough resolution for your case.

It might take a while and incur in AWS DataTransfer cost on you if you run this locally away from AWS, so you ideally want to run this from an EC2 instance in the same VPC.

If you are anyway bold to run locally away from AWS with this approach even with its costs, you might want to redirect stdout and stderr to some docs in order to check them later if necessary, instead of running the costly command line again.

Wrapping up, I might go with:

grep -r Key /PATH/TO/MOUNT/POINT/ | \
        grep 951332 \
        >/LOCAL/PATH/TO/grep_stdout \
        2>/LOCAL/PATH/TO/grep_stderr
#   In /LOCAL/PATH/TO/grep_stdout should be the paths to the docs you
# were searching.

Alternatively:

grep -r Key /PATH/TO/MOUNT/POINT/ | \
        grep 951332 &>/LOCAL/PATH/TO/all_grep_outputs
#   In /LOCAL/PATH/TO/all_grep_outputs should be the paths to the docs you
# were searching.
-1

Cloudgrep is good for this:

./cloudgrep --bucket test-s3-access-logs --query 9RXXKPREHHTFQD77
1
  • 6
    Can you modify this to make it more relevant to the question, e.g., by using the search terms and bucket names mentioned?
    – Kusalananda
    Commented Nov 11, 2023 at 6:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .