Use case: Let's say there is a folder full of large RDF+XML files and they follow different namespaces. We want to find those files that follow a given namespace (for example xmlns:crm="http://www.cidoc-crm.org/cidoc-crm#").

Scanning files from top to bottom is time-consuming as a namespace is always defined in the first few lines of an XML file.

How to pick out files in a massive folder in which the header contains specific text?


The answer looks something like this:

 awk '/hi/ { if (FNR >= 5) { nextfile }; print FNR, FILENAME }' *

Change the /hi/ with your regex matching string. Change the 5 to be the last line number in the header you want to look at.

  • You've got your logic backward. You're only checking for FNR > 5 in lines that match the pattern. Nov 25 '14 at 15:11
  • nextfile requires GNU awk or recent versions of mawk. Nov 25 '14 at 15:13
  • Interesting, I was looking at this http://www.manpagez.com/man/1/awk/ and it has nextfile. Will fix readability.
    – sparticvs
    Nov 25 '14 at 15:24
  • Indeed. While nextfile was introduced by GNU awk, Brian Kernighan (the k in awk) added it in his implementation in 1996, and that eventually made its way to the BSD awks. Which means nextfile is more widely available than I initially thought (still not POSIX or available in many commercial Unices though). Nov 25 '14 at 15:47

To look in the first 5 lines only, if your awk supports nextfile:

TEXT='xmlns:crm="http://www.cidoc-crm.org/cidoc-crm#"' \
  find . -type f -exec awk '
     index($0, ENVIRON["TEXT"]) {print FILENAME; nextfile}
     FNR == 5 {nextfile}' {} +

(if your awk doesn't support nextfile, that will be silently ignored and the files will be read fully).

Or you could use perl:

TEXT='xmlns:crm="http://www.cidoc-crm.org/cidoc-crm#"' \
  find . -type f -exec perl -Tne '
     if (index($_, $ENV{"TEXT"}) >= 0) {print "$ARGV\n"; close ARGV}
     elsif ($. == 5) {close ARGV}' {} +

If all your files have a namespace declaration, With GNU/FreeBSD/OSX grep or any implementation that has the -m option, you can look for the namespace declaration and quit as soon as it's found, then pipe the result into a second grep to get the namespace that you're interested in.

grep -m xmlns: *.xml  |grep 'xmlns:crm="http://www.cidoc-crm.org/cidoc-crm#"'

If some files lack a recognizable namespace declaration or some other pattern that says “quit”, use sed or awk instead to specify a quitting condition based on the line number. With sed, there's no way to count lines per file, so you'll need to invoke sed on each file:

for x in *.xml; do
  if [ -n "$(sed -n -e 's/.*xmlns:crm="http:\/\/www.cidoc-crm.org\/cidoc-crm#".*/1/' -e '/xmlns:/q' -e '5q')" ]; then
    printf '%s\n' "$x"

See Stéphane Chazelas's answer for an awk solution.

Any method based on text processing tools is inherently fragile. For example, it would pick up a namespace declaration that's commented out. Unless you're going to review the results manually or you know that all your files have a “tame” structure (no comments, no payload text that looks like what you're searching, etc.), you should use a proper XML parser instead — but it will be slower.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.