0

Is there a way to find get all the lines in a file that are prefixes of any of the lines in another?

Using grep I can check if a single pattern is a prefix of any of the lines. For instance if file1 is

1
2
3

and file2 is

100
10
25
55

grep '^1' file2 will give me

100
10

Is there a way to loop through the lines in file1 and check if those lines are prefixes of the lines in file2? In this case if I check for prefixes from file1 in file2 the output will be

100
10
25
4

Turn your file1 into a list of proper regular expressions:

sed 's/^/^/' file1 >patterns

This simply inserts a ^ as the first character on each line.

Then use that with grep on your other file:

grep -f patterns file2

Testing:

$ sed 's/^/^/' file1 >patterns
$ cat patterns
^1
^2
^3
$ grep -f patterns file2
100
10
25

Without a temporary pattern file:

With a shell that understands process substitutions:

grep -f <( sed 's/^/^/' file1 ) file2

In a shell that don't have process substitutions:

sed 's/^/^/' file1 | grep -f /dev/stdin file2

With the process redirection facility of the yash shell (which uses the same syntax as the above process substitution, but with different semantics):

grep -f /dev/stdin file2 <( sed 's/^/^/' file1 )
  • Works really well. Although for large files it is really slow (as expected). Are there any faster alternatives? Perhaps something that can be applied after sorting the files? – Unni Jan 30 at 12:27
  • 1
    @Unni I would rather question if this is the right way of solving whatever underlying issue that you are dealing with. There may be better ways of doing it, depending on what it is that you're actually doing. Note also that if you have patterns that are sub-patterns of each other, such as ^11 and ^1, then the longer pattern can be removed. – Kusalananda Jan 30 at 12:29
  • @Unni How big is your pattern file (file1)? – Kusalananda Jan 30 at 12:38
  • The file contains more than 1.7M patterns and I am looking them up against 4.5M strings. – Unni Jan 30 at 22:41
  • 2
    Use this approach on the patterns file itself to see whether any of the patterns are in fact prefixes of other patterns. :) – Jim L. Jan 31 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.