1

I have around 100 66 Gigabyte huge "log.txt" (log1.txt,log2.txt,log3.txt and so on) files which I need to search for different Strings.

I have all 42.000 strings in a text file.

Right now, I'm using grep (grep -F -f strings.txt log.txt) but it takes around 50 minutes to search one log.txt.

Do you know a faster way to search all files? After searching all 100 Files, a new set of 100 files will be generated for searching.

9
  • what's the naming convention of your log files? Nov 24, 2017 at 16:14
  • What do you mean?
    – Daniel
    Nov 24, 2017 at 16:17
  • what do you mean with your "log.txt" Files? Files - is plural, log.txt is one single name. Elaborate your question, be more exact Nov 24, 2017 at 16:19
  • Each file is named like log1.txt, log2.txt, log3.txt an so on.
    – Daniel
    Nov 24, 2017 at 16:22
  • 2
    This are not just "annoying" questions from me. I have already realized a potential working solution, but you should present the main goal in specifics. Nov 24, 2017 at 16:32

1 Answer 1

1

Running the danger of being accused of product promotion, but if this problem is structural (as in daily time consuming) I would recommend Elasticsearch.

How they do it I don't know but I got 80GB files imported daily that elastic returns answers from in miliseconds.

Disclaimer: I am not related to that company, do not benefit in any way. Just a happy user of their products.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .