I have a file on the form

2e95d7582c53583fa8afb54e0fe7a2597c92cbba 1461065389 52880 temp/hello/file.txt
46c897a7aa8a641f46080b3431860bd0cd4a8f05 1461066221 207 temp/Another file.txt
83c8ce6b163ec1c615617fa0dbde9e928bc3daf4 1461056193 86112 Pictures/a photo.jpg

That is, each line has a 40-character long hex number, followed by space, an integer, space, an integer, space, and a file path that may contain spaces. Each path is unique.

In a bash script I have variables on the same form as a line in the file, only without the first hex string and space (41 first characters), e.g.:

myvar="1461066221 207 temp/Another file.txt"

My goal is to find the line in the file that matches myvar exactly when the first 41 characters are ignored. If such a match is found I would like the variable line to be set to the full line in the file. For the example above, line would be set to

46c897a7aa8a641f46080b3431860bd0cd4a8f05 1461066221 207 temp/Another file.txt

If there is no such match, line should be set to the empty string, or left unset.

My solution is this (filelist is the name of file):

line=$(grep --color=never -E "^.{41}$myvar$" $filelist)

A problem is that when $myvar is expanded it might contain special symbols like ., +, even ^ or $, etc, which have a special meaning for grep. I want grep to perform an exact match of myvar on all but the first 41 characters of a line.

  • Can the variable only match the end of the string? If so, you could use grep's -F option. You just can't specify "at the end" with that.
    – Kevin
    Apr 19 '16 at 16:58
export myvar
awk 'substr($0, 42) == ENVIRON["myvar"]' < "$filelist"
  • Given the concise, and for me very understandable, solution I choose this as the answer. Thanks.
    – DustByte
    Apr 20 '16 at 15:11

Perl to the rescue!

perl -ne 'BEGIN { $search = shift }
          print if /^.{41}\Q$search\E$/;
         ' -- "$myvar" "$filelist"
  • -n reads the file line by line.
  • The BEGIN block retrieves the $myvar from the first argument into Perl variable $search.
  • \Q...\E quotes the inner part (see quotemeta). This handles all the special characters the variable can contain. Don't forget to double quote the shell variables!
  • 1
    Oh no, not Perl! ;) My whole life I've successfully avoided Perl. Anyway, cheers! I've tested your solution and it seems robust! I will adopt it into my script for now.
    – DustByte
    Apr 19 '16 at 16:05
  • 3
    @DustByte: Your life might have been much easier ;-) Don't listen to haters, use the right tool for the job!
    – choroba
    Apr 19 '16 at 16:06

This is a different way to go; it first sets 'line' to be empty, and sets it if it finds that 'myvar' matches the trailing 3 fields.

while IFS=' ' read -r hex int1 int2 rest
  if [[ "$myvar" = "$int1 $int2 $rest" ]]
    line="$hex $int1 $int2 $rest"
done < filelist

Here's yet another way, using bash's mapfile builtin and second temporary associative array:

mapfile -t < filelist # sets MAPFILE array
declare -A temparray
shopt -s extglob
for val in "${MAPFILE[@]}"
  short="${val##*([^ ]) }"
unset -v MAPFILE temparray val short

This brings the file in to an indexed array MAPFILE, which we then loop over. The loop sets the "short" variable to the "suffix" part of the line -- it strips out leading values that match the pattern "zero or more non-spaces followed by a space", then sets the associative array value for "$short" to be the whole line ("$val"). Doing this sort of matching with the * requires shopt -s extglob to be in effect. We then set line to be the resulting value (or if that value doesn't exist in the array, bash will return the empty/null value).

  • I like the idea of reading the file into an associative array. I can imagine this will speed up the look-ups. And yes, I do intend to do many of those! However, I tried your script on my file (around 80,000 lines) and after half an hour of all fans at high speed, I interrupted. The for-loop seems to take very long.
    – DustByte
    Apr 20 '16 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.