When writing a shell script, it is best to specify the verified variables first, and filenames last, so you can vary the number of files specified. In your case, you have the column number, a file with the patterns in it, and two (or perhaps more) file names to work on. So, start your Bash script with
#!/bin/bash
if [ $# -lt 2 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
echo ""
echo "Usage: $0 [ -h | --help ]"
echo " $0 COLUMN PATTERNFILE [ FILE(s) ... ]"
echo ""
exit 0
fi
The if
clause above uses old-style POSIX shell formatting, and will work in dash
(and other POSIX shells) as well as in most old-style sh
shells as well. The intent is that if the user does not specify any command line arguments, or just a -h
or --help
, the script just prints a short help text.
You should expand on the help text, by the way, because it makes it much easier to find out what it does in two or three months, after you've forgotten you wrote it. (Happens to me all the time, and I have lots of such scriptlets, so I've found this practice well worth the little effort.)
Next, extract the required parameters (only one, above), and shift
them out, so that we can use "$@"
to refer to all file names specified on the command line:
column=$1
patternfile="$2"
shift 2
Note that I like to put double quotes around stuff I want expanded in the shell, even when not explicitly necessary. This is because most real-life problems I encounter with shell scripts are due to forgetting quoting an expansion, when it would have been necessary. This practice is easy to remember, and other than having some know-it-all comment that "you don't actually need those double-quotes there" in an annoying nasal tone, they do no harm.
Lets then use awk
to process the input files:
awk -v column=$column \
'BEGIN {
RS = "[\t\v\f ]*(\r\n|\n\r|\r|\n|)[\t\v\f ]*"
FS = "[\t\v\f ]*;[\t\v\f ]*"
}
The backslash at the end of the first line above just tells the shell that the command continues on the next line. Also note that there is no closing single quote '
, so the lines below are actually continuation to the command-line string parameter we are supplying to awk
.
The BEGIN
rule in awk is executed before the files are processed. The above RS
sets the record separator to any newline convention, and includes any leading or trailing whitespace on each line. Similarly the field separator is a semicolon, but including any whitespace surrounding it. Thus, a ; b
has two fields, the first being a
and second b
, neither having any whitespace.
I use the following idiom to keep track of which input file is being processed:
FNR==1 { ++filenum }
If just means that for the very first record in each input file we process, we increment the filenum
variable. Incrementing an uninitialized variable is the same as incrementing a zero, so we get 1
for the first input file, and so on.
We want to just remember the contents of each line in the first input file, our pattern file:
filenum==1 { pattern[$0] }
Awk arrays are associative, so we can just use an associative array to hold the known patterns. Above, we use a funny awk feature to our advantage: If you try to access an associative array entry that does not exist yet, awk creates it!
For the rest of the files, we just check if the field $column
(supplied to the awk scriptlet in awk variable column
) matches (exactly) any of the patterns seen in the first file, and if so, we print the entire record:
filenum > 1 && ($column in pattern) { printf "%s\n", $0 }
Above, $column
has a different meaning compared to a shell script. Here, column
is a variable, and $column
expands to the value of the column
'th field in the current record (zeroth column being the entire record, however). The foo in array
syntax is awkism for checking if array
contains a key foo
. So, overall, for the second and further input files, if the column
'th field value was listed in the first input file, the record is printed. to standard output.
We are still in the awk
command-line parameter string, and need to close the single-quoted string. We also want to supply it with the file names:
' "$patternfile" "$@"
which concludes this awk scriptlet.
awk
) only as a tool to extract data and do the actual processing in a language that is absolutely crap for text processing (shell). I suggest you edit your question, show us an example of your input files and your final desired output and ask for help on doing the actual processing rather than this, almost certainly unnecessary, preprocessing step. – terdon♦ Dec 6 '16 at 10:16;
? By the way, are you just looking forgrep -FLf patterns.txt file1.csv file2.csv
? – terdon♦ Dec 6 '16 at 10:49