1

I want to grep a pattern across 10M files really fast in a 36 core machine i tried this

find . -name '*.xml' -type f | xargs  -P 20 grep "username" >> output

But i am getting some other results in between.

Is there any better way to do this ?

Thanks in advance.

  • What are the "other" results and what are you looking for? The line content? The filename? What is your expected output? – bu5hman Nov 8 at 12:04
  • You're working on XML files? What is it that you are trying to extract from these? Can you give examples of actual data? XML documents are best parsed using XML parsers and queried using e.g. XPath queries, not by line-oriented tools like grep. – Kusalananda Nov 8 at 12:10
  • @bu5hman i mean result not matching with my pattern – user3398900 Nov 8 at 12:13
  • CPU is likely not the bottleneck if you're grepping for fixed strings. So parallelising is more likely to harm than help. See also Grep in couple thousands files and Recursive grep vs find / -type f -exec grep {} \; Which is more efficient/faster? – Stéphane Chazelas Nov 8 at 13:09
  • How are files stored? Redundant RAID arrays? HDDs or SSDs? Can you arrange for the output to be stored on different HDDs? – Stéphane Chazelas Nov 8 at 13:17
1

Given that your data is on non-RAIDed HDDs, I doubt you'll get better performance from parallelizing, the bottle neck is most likely to be I/O, not CPU.

LC_ALL=C grep -rwF --include='*.xml' username . > /on/some/other/disk/output

May be close to the best you can achieve.

To parallelize, you'd want to do it as:

LC_ALL=C find . -name '*.xml' -type f -print0 |
  LC_ALL=C xargs -r0P20 -n 1000 grep -HFw --line-buffered username > output

assuming there's no output line (input line + file pathname) longer than 4KiB, and note the lines of all 20 concurrent greps will end up interleaved.

See:

for details.

0

I think it's because of the pattern being too general. Usually the command line utilities print errors to the stderr or the terminal. They shouldn't get in the output file.

  • Ok updated the one i have tried is there any better way to do it ??? – user3398900 Nov 8 at 12:18
0

If you are grepping xml files in this way then your search will return the entire line containing the search string and, if the xml file has no newlines, the entire file contents. Quite a lot of "other" across 10M files.

As per @Kusalananda comment it isn't good practice to brute force an xml with grep and an xml parser e.g. xmllint is a better tool, however, if you insist ......

Check man for grep and read up on the -o option to restrict the returned value and use regex that defines the whole length of the match you are looking to find.

If username is an attribute

grep -o 'username="[^"]*"'

Or better

xmllint --xpath "//@username"

If username is a node then something like

grep -o "username>[^<][^<]*"

Or better

xmllint --xpath "//username"

With either of the xmllint queries, just wrap the query in string() to extract the attribute or node text.

xmllint --xpath "string(//username)"
xmllint --xpath "string(//@username)"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.