Using grep with a pattern that contains a variable to be expanded

Question

I have a file on the form

2e95d7582c53583fa8afb54e0fe7a2597c92cbba 1461065389 52880 temp/hello/file.txt
46c897a7aa8a641f46080b3431860bd0cd4a8f05 1461066221 207 temp/Another file.txt
83c8ce6b163ec1c615617fa0dbde9e928bc3daf4 1461056193 86112 Pictures/a photo.jpg
...

That is, each line has a 40-character long hex number, followed by space, an integer, space, an integer, space, and a file path that may contain spaces. Each path is unique.

In a bash script I have variables on the same form as a line in the file, only without the first hex string and space (41 first characters), e.g.:

myvar="1461066221 207 temp/Another file.txt"

My goal is to find the line in the file that matches myvar exactly when the first 41 characters are ignored. If such a match is found I would like the variable line to be set to the full line in the file. For the example above, line would be set to

46c897a7aa8a641f46080b3431860bd0cd4a8f05 1461066221 207 temp/Another file.txt

If there is no such match, line should be set to the empty string, or left unset.

My solution is this (filelist is the name of file):

line=$(grep --color=never -E "^.{41}$myvar$" $filelist)

A problem is that when $myvar is expanded it might contain special symbols like ., +, even ^ or $, etc, which have a special meaning for grep. I want grep to perform an exact match of myvar on all but the first 41 characters of a line.

Answer 1

Perl to the rescue!

perl -ne 'BEGIN { $search = shift }
          print if /^.{41}\Q$search\E$/;
         ' -- "$myvar" "$filelist"

• -n reads the file line by line.
• The BEGIN block retrieves the $myvar from the first argument into Perl variable $search.
• \Q...\E quotes the inner part (see quotemeta). This handles all the special characters the variable can contain. Don't forget to double quote the shell variables!

Answer 2

export myvar
awk 'substr($0, 42) == ENVIRON["myvar"]' < "$filelist"

Answer 3

This is a different way to go; it first sets 'line' to be empty, and sets it if it finds that 'myvar' matches the trailing 3 fields.

line=
while IFS=' ' read -r hex int1 int2 rest
do
  if [[ "$myvar" = "$int1 $int2 $rest" ]]
  then
    line="$hex $int1 $int2 $rest"
  fi
done < filelist

Here's yet another way, using bash's mapfile builtin and second temporary associative array:

mapfile -t < filelist # sets MAPFILE array
declare -A temparray
shopt -s extglob
for val in "${MAPFILE[@]}"
do 
  short="${val##*([^ ]) }"
  temparray[$short]="$val"
done
line=${m[$myvar]}
unset -v MAPFILE temparray val short

This brings the file in to an indexed array MAPFILE, which we then loop over. The loop sets the "short" variable to the "suffix" part of the line -- it strips out leading values that match the pattern "zero or more non-spaces followed by a space", then sets the associative array value for "$short" to be the whole line ("$val"). Doing this sort of matching with the * requires shopt -s extglob to be in effect. We then set line to be the resulting value (or if that value doesn't exist in the array, bash will return the empty/null value).