How to search for a string inside a lot of .gz files in Amazon S3 bucket subfolder? I tried to mount it via s3fs and zgrep but it's sooooo slow. Do you use any other methods?

Maybe is there any Amazon service which I could use to quickly zgrep them?


I find the quickest way is to copy them locally first then do a local zgrep:

aws s3 cp s3://bucket/containing/the/logs . --recursive

This will copy (cp) all the logs to your current directory (.) and include all sub folders too (--recursive).

Then a local zgrep:

zgrep "search words" *.gz

Or to recursively search sub directories too:

find -name \*.gz -print0 | xargs -0 zgrep "STRING"

(Taken from unix.stackexchange.com.)

  • Unfortunatelly, as I mentioned in my question, it's too slow (a lot of files to check). I'm looking for faster solution. – Michal_Szulc Sep 26 '16 at 21:09
  • There is a big difference between searching locally on a laptop with ssd and an network mounted partition. When I get a minute I will do a comparison. @Michal_Szulc – geedoubleya Sep 28 '16 at 9:26
  • I tried also copying and zgreping all resourcese to local machine with ssd (using your script)), but this solution requires a lot of disk space. that's why i'm looking for some kind of remote solution (using api or any other AWS service, which will be either fast or do not require a lot of diskspace). – Michal_Szulc Sep 28 '16 at 9:52

It's not grep, but you can now query logs with Athena:

First, create a table from your S3 bucket:

  BucketOwner string,
  Bucket string,
  RequestDateTime string,
  RemoteIP string,
  Requester string,
  RequestID string,
  Operation string,
  Key string,
  RequestURI_operation string,
  RequestURI_key string,
  RequestURI_httpProtoversion string,
  HTTPstatus string,
  ErrorCode string,
  BytesSent string,
  ObjectSize string,
  TotalTime string,
  TurnAroundTime string,
  Referrer string,
  UserAgent string,
  VersionId string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
 'serialization.format' = '1',
  'input.regex' = '([^ ]*) ([^ ]*) \\[(.*?)\\] ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) \\\"([^ ]*) ([^ ]*) (- |[^ ]*)\\\" (-|[0-9]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) (\"[^\"]*\") ([^ ]*)$'
LOCATION 's3://s3-server-access/logs/'

Then, you can query it with SQL:

SELECT requestdatetime, bucket, remoteip, requesturi_key
FROM s3accesslogs
WHERE bucket IN ('bucket1', 'bucket2')
    AND remoteip = ''
ORDER BY requestdatetime DESC
LIMIT 100;
  • Matton I'm afraid that my previous comment: unix.stackexchange.com/questions/312436/… also fits to your solution. Still searching for a solution! – Michal_Szulc Oct 18 '17 at 20:13
  • Yeah, I'm hitting some painfully slow queries as well. They don't allow indexing either. This might be a use-case for Redshift or something, but I'm not sure yet. – Curtis Mattoon Oct 19 '17 at 19:16

I had the same problem. I tried writing a python script to download the file and run it through zgrep. But it took 30 sec just to run one grep command. Files were also around 200MB, so overall time was very high. And I had to do it for hundreds of files.

In my case, I wrote a lambda function and ran it parallely for different .gz files. Use the /tmp storage to download the .gz file for a lambda instance and run zgrep.

os.chdir('/tmp/') os.mkdir('s3_logs') os.chdir('s3_logs') s3.download_file(Bucket, s3object, '/tmp/s3_logs/file.gz') query = 'zgrep String file.gz' result = os.popen(query).read() print(result)

Multiple instances of lambda can make it at least some times faster. Although Remember that the tmp storage for lambda has only 500MB storage.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.