I need to process a large subset of a large set of files with
AWK(*) so that it accumulates a set of variables across the files.
The straighforward approach of passing multiple filenames to
AWK with a file wildcard worked fine for a small fileset, but expectedly results in
"Argument list too long" when run with a production-sized set of files.
What is the best-practice approach to such a problem?
the entire set of files is 20-50K files; a subset for a single run is 5-10K for now (but great if it could scale easily)
I need to count occurrences of each word across a set of files, giving each file a runtime-defined weight: each word in the same file gets the same weight, but the same word occurring in different files get different weight. For each word, file weights are then added.
therefore splitting the fileset into smaller subsets would mean aggregating intermediary results. It doesn't look very elegant, and will require to add floating points while joining several intermediary files, which makes the whole procedure even less readable and intuitive.
another approach I can think of is to feed
awkwith an output of
cat. What I don't like is sacrificing readability of
ENDFILEand working around with parsing some delimiter between files to reset file-specific weight, counters and arrays.
file subset to process from the current folder is provided as a separate file A; in
BEGINFILEsection I skip files that I don't need
- weight for each file X is derived from a combination of that file with a reference file B; basically it's a ratio of words common between X and B to the number of words in X
- separating file weight calculation from aggregating across files would mean two read passes across dozens of GB, which I would like to avoid
(*) Or maybe
AWK is not the best tool for such processing? If so, what alternative would you recommend?