I have a shell script that runs regularly and the following part of it causes a slowdown.

grep -v -f RemoveTheseGoodIPs.txt FromTheseShadyIPs.txt > RemainingBadIPs.txt

It works. It just takes 156 seconds to give me an output. I'm hoping to figure out a quicker way to process that still is easy to understand and elegant.

For context: The "FromTheseShadyIPs.txt" is a list of 200k shady IP addresses and the "RemoveTheseGoodIPs.txt" is a whitelist file of 3k IPs that are good. Ultimately, I'm producing a blacklist that my external firewall can reference but I don't want my 3k good IPs in my blacklist. If it helps, the order of the IPs in either file doesn't matter and they're already deduplicated within each file. Processing server runs Debian 9 on decent specs.

  • Apart from grep you could use diff as described here: unix.stackexchange.com/questions/56625/… But I agree with Anthony that your problem unlikely this grep command. I tried it with 200k random IPs and 3k to filter. Took only 3.3s with your commands and not even 100ms with -F flag. Commented Jul 29, 2023 at 18:27
  • grep -F can run faster than grep if you're able to provide fixed strings instead of regular expressions. Commented Jul 29, 2023 at 20:02

2 Answers 2


Try adding the -F option. It does not do any regexp processing and just interprets input as string literals.

  • 1
    I tried it with 200k IPs filtered with a 3k IPs list. With -F flag <100ms, without ~3.3s. So the difference is significant. Commented Jul 29, 2023 at 18:28
  • 1
    Nothing fancy. Didn't follow all the rules for good performance measurements but with such a large gap... Used the time "command". Bash docs have some details. To generate some test data I used for ((i = 0; i < 10; i++)); do echo "$(($RANDOM % 256)).$(($RANDOM % 256)).$(($RANDOM % 256)).$(($RANDOM % 256))"; done. Commented Jul 29, 2023 at 19:43
  • 2
    WOW! Adding the -F took my time from ~150seconds to 62ms!!! (Yes, a lower-case ms!) Commented Jul 29, 2023 at 20:39
  • 1
    @AnthonyDeVellis here's how I measured it: starttime=date +"%s%N"; commands to measure; finishtime=date +"%s%N"; /usr/bin/printf "%'d\n" $(($finishtime - $starttime)); That produces the delta in time measured in nanoseconds. Commented Jul 29, 2023 at 20:39
  • 2
    Having seen this kind of timing discrepancy between grep usage in Solaris vs Linux with -F, I am reasonably convinced that GNU/grep with -F makes a hash lookup (which it can do with fixed strings, but not with REs). On Solaris, I replaced a grep -F of 16,000 strings in 10 million lines (taking 4 hours) with an awk hash (taking 100 seconds). Commented Jul 29, 2023 at 22:18

There's more important issues with your script than execution speed, it'll also encounter false matches in 2 ways:

  1. Regexp vs String: You're using a regexp comparison when you should be using a string comparison. As written the .s from the IP addresses in RemoveTheseGoodIPs.txt will match any character FromTheseShadyIPs.txt, and
  2. Partial vs Full: You're using a partial line comparison when you should be using a full-line comparison. As written a shorter IP address in RemoveTheseGoodIPs.txt will match any IP address that includes it in FromTheseShadyIPs.txt

Given that, your current script is almost certainly removing IP addresses from FromTheseShadyIPs.txt that are not present in RemoveTheseGoodIPs.txt, thereby effectively breaking your firewall.

For example, if RemoveTheseGoodIPs.txt contained and FromTheseShadyIPs.txt contained 911.253.456.789 then your grep would remove that 2nd IP address because you're doing a partial-line regexp match instead of the full-line string match which you need:

$ head RemoveTheseGoodIPs.txt FromTheseShadyIPs.txt
==> RemoveTheseGoodIPs.txt <==

==> FromTheseShadyIPs.txt <==

$ grep -v -f RemoveTheseGoodIPs.txt FromTheseShadyIPs.txt

You should be using

$ grep -vFxf RemoveTheseGoodIPs.txt FromTheseShadyIPs.txt

to make your script work. That's -F for string instead of regexp comparison, and -x for full-line instead of partial comparison. That will probably also be faster than your current script but the far more important difference is it'll work robustly.

If your grep doesn't support any of those options and you can't get a version that does then use the following with any awk instead:

$ awk 'NR==FNR{a[$0]; next} !($0 in a)' RemoveTheseGoodIPs.txt FromTheseShadyIPs.txt

As @Paul_Pedant mentions in a comment using awk may be faster than grep for this anyway, depending on your grep and awk implementations.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .