1

I need to process a large subset of a large set of files with AWK(*) so that it accumulates a set of variables across the files.

The straighforward approach of passing multiple filenames to AWK with a file wildcard worked fine for a small fileset, but expectedly results in "Argument list too long" when run with a production-sized set of files.

What is the best-practice approach to such a problem?


Some details:

  • the entire set of files is 20-50K files; a subset for a single run is 5-10K for now (but great if it could scale easily)

  • I need to count occurrences of each word across a set of files, giving each file a runtime-defined weight: each word in the same file gets the same weight, but the same word occurring in different files get different weight. For each word, file weights are then added.

  • therefore splitting the fileset into smaller subsets would mean aggregating intermediary results. It doesn't look very elegant, and will require to add floating points while joining several intermediary files, which makes the whole procedure even less readable and intuitive.

  • another approach I can think of is to feed awk with an output of find & cat. What I don't like is sacrificing readability of BEGINFILE/ENDFILE and working around with parsing some delimiter between files to reset file-specific weight, counters and arrays.

  • file subset to process from the current folder is provided as a separate file A; in BEGINFILE section I skip files that I don't need

  • weight for each file X is derived from a combination of that file with a reference file B; basically it's a ratio of words common between X and B to the number of words in X
  • separating file weight calculation from aggregating across files would mean two read passes across dozens of GB, which I would like to avoid

(*) Or maybe AWK is not the best tool for such processing? If so, what alternative would you recommend?

  • You're running into a system limit (see xargs --show-limits). A Perl script could be passed the directory and use perl to read the directory and process each of the files. – waltinator Aug 23 '18 at 0:21
0

One option, if your filenames contain no quotes or whitespace, would be to pile them together with cat:

printf '%s ' * | xargs cat | awk ...

The above simply gets around the "argument list too long" error by using a builtin (printf) to print every filename, which is then sent to xargs, which breaks the filenames up into batches that it then sends to cat, the output of which is then sent to awk.

But: don't use xargs

If you have GNU awk available (gawk) at version 4.1 or above, where dynamic module loading was introduced, it contains an extension that can read a directory itself, bypassing the problem.

Here's a sample gawk program that will open and read the files in whatever directory you pass to it; you then have to explicitly read from each file that you're interested in. The benefit is that you have a single (GNU) awk program that will read every file.

@load "readdir"
@load "filefuncs"

BEGIN { FS = "/" }
{
        result = stat($2, statdata)
        if (statdata["type"] != "file")
                next
        FS = " "
        while(getline < statdata["name"] > 0) {
                #print $1
        }
        FS = "/"
}

The script's main loop goes through every argument given on the command-line and attempts to open it as a directory. The resulting fields are:

  • $1 = inode number
  • $2 = file name
  • $3 = type of file

We then use the filefuncs function stat to check the type of the file. If it is not a plain file, we skip it. Otherwise, we set FS back to the normal value and use getline to read through the file. After we're done with each file, we reset FS back to / so that it can split the next filename from readdir.

I learned about gawk's readdir here and about gawk's filefuncs stat here.

  • 1. Do I understand correctly that xargs way gives a single concatenated "file" to AWK, leaving it up to AWK to detect file boundaries? – wass rubleff Aug 23 '18 at 10:04
  • 2. How traditional multiple blocks of per-line statements, with a pre-condition before each, should be restructured for this approach? Same question about gawk's BEGINFILE/ENDFILE blocks. – wass rubleff Aug 23 '18 at 10:08
  • Correct; awk would see one continuous input stream, with no file boundaries at all. If the files had internal markers, you could detect changes that way. – Jeff Schaller Aug 23 '18 at 10:08
  • 2. You’d have to do your own pattern matching of lines inside the while loop; begin/end file would be immediately before and after that while loop. – Jeff Schaller Aug 23 '18 at 10:10
  • 2: How do I iterate over lines of each file? – wass rubleff Aug 23 '18 at 10:12
0

If the arguments are too many, you'll have to open and process the files yourself. With awk, without using any extensions you can use this (same idea as Jeff's answer):

awk '{ filename = $0; while(getline < filename > 0) { print $0; }}'

E.g., combine with a find command to find the files you need:

find /etc/ -maxdepth 1 -type f -perm -444 -size 1 | \
  awk '{ filename = $0; while(getline < filename > 0) { print filename ":" $0; }}'

Further, depending on the version of awk, it is possible to push more files to be processed as documented here.

A program can alter ARGC and the elements of ARGV. Each time awk reaches the end of an input file, it uses the next element of ARGV as the name of the next input file. By storing a different string there, a program can change which files are read. Use "-" to represent the standard input. Storing additional elements and incrementing ARGC causes additional files to be read.

To illustrate with an example:

find /etc/ -maxdepth 1 -type f -perm -444 -size 1 | \
  awk '
    # When reading from STDIN, assume it is a list of files to read
    FILENAME == "-" { ARGV[ARGC] = $0; ARGC += 1 }
    # When not reading STDIN, it is a file to process
    FILENAME != "-" { print "---", FILENAME ":" FNR ":" $0; }
    # These will run after every file, including STDIN, hence the check
    BEGINFILE { if (FILENAME != "-") { print ">>>", FILENAME; } }
    ENDFILE   { if (FILENAME != "-") { print "<<<", FILENAME, FNR, "lines"; } }'
  • 1. Is it a single AWK process for the entire file set? – wass rubleff Aug 23 '18 at 9:59
  • 2. How traditional multiple blocks of per-line statements, with a pre-condition before each, should be restructured for this approach? Same question about gawk's BEGINFILE/ENDFILE blocks. – wass rubleff Aug 23 '18 at 10:01
  • @wassrubleff, updated the answer to show how to do it with BEGINFILE. – chutz Aug 24 '18 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.