I have around 100 66 Gigabyte huge "log.txt" (log1.txt,log2.txt,log3.txt and so on) files which I need to search for different Strings.

I have all 42.000 strings in a text file.

Right now, I'm using grep (grep -F -f strings.txt log.txt) but it takes around 50 minutes to search one log.txt.

Do you know a faster way to search all files? After searching all 100 Files, a new set of 100 files will be generated for searching.

  • what's the naming convention of your log files? Nov 24 '17 at 16:14
  • What do you mean?
    – Daniel
    Nov 24 '17 at 16:17
  • what do you mean with your "log.txt" Files? Files - is plural, log.txt is one single name. Elaborate your question, be more exact Nov 24 '17 at 16:19
  • Each file is named like log1.txt, log2.txt, log3.txt an so on.
    – Daniel
    Nov 24 '17 at 16:22
  • 2
    This are not just "annoying" questions from me. I have already realized a potential working solution, but you should present the main goal in specifics. Nov 24 '17 at 16:32

Running the danger of being accused of product promotion, but if this problem is structural (as in daily time consuming) I would recommend Elasticsearch.

How they do it I don't know but I got 80GB files imported daily that elastic returns answers from in miliseconds.

Disclaimer: I am not related to that company, do not benefit in any way. Just a happy user of their products.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.