don't do this
... at least not for this particular DNS request. Why not? Well, first, you're not really making it easier for yourself, your CPU, and you're making it much worse for the client. Nearly every browser on every platform when given a DHCP address uses 'automatic proxy detection'. This auto detection works by looking for a DNS entry for a hostname
wpad.subdomain.subdomain.currentdomain.xx proceding down to
wpad.currentdomain.xx. Second, the browser requests the OS to get the DNS, and the OS (subject to various TCP / UDP parameters) will keep trying the request to the DNS server until it gets an answer. When you block the input, you prevent your DNS server from giving any answer at all, so the client tries again and again, and again. And by blocking this, you're not somehow "training" the client to stop sending the requests. If it doesn't get an answer, it simply stops trying to use a proxy and uses the direct connection. Meanwhile the user is just waiting there for the browser to try the direct connection. She might be waiting there for 15 seconds before the browser becomes responsive. This mechanism is now built into all Android and iOS devices, so every iPad, AppleTV, Samsung smartphone is taking a long time to connect to the internet. So basically, you're bring a dick.
How to do it (optimally, correctly)
But let's say you're going to block a DNS query based on the QNAME. The QNAME comprises a set of "labels". While we see the labels in dotted notation, in the query packet, each label is prefixed by its length in bytes, all of which are terminated by a NUL. 1
1The OP figured this part out already, but made it sound like it had something to do with the hex-string.
The hex-string, as I confirmed by looking at the source to iptables
1.4.9, since no manual I could find adequately describes its behavior, is of the (quasi BNF) form
HEXSTRING := SUBSTRING [ SUBSTRING ... ]
SUBSTRING is one of
'|' HEXDIGIT HEXDIGIT [ SPACE ] '|'
where CHAR is pretty much anything the input can handler, and HEXDIGIT is what isxdigit() says it is, with the combination converted via
sscanf(s,"%x",buf). The escaping character ensures that one can bypass "hex mode" and match the literal string "|". Otherwise, it does nothing special. The trailing
| is not strictly necessary if the present SUBSTRING is the last in HEX-STRING.
Next, optimizing the search: the Query section starts no earlier than the 13th byte of the DNS request, which means starting from byte 40 ( 20 (IP) + 8 (UDP) + 13 - 1 ). If you can test that the packet is a Query, which you should, then you don't need to worry about a limit to search for -- just search to the end of the packet. To test if it's a query, test the high 5 bits of of byte 31 in the packet. You might also want to take into account the slight difference the IP header, which in rare conditions is 24+ bytes instead of 20, making the byte to test 35 (or 39).
Optimize further by using the
bm (Boyer-Moore) algorithm. There's nothing wrong with the
kmp, but in general, Boyer-Moore is faster and uses fewer resources.
Tying it all together, we should have something like:
-A INPUT -s 126.96.36.199/32 -p udp -m udp --dport 53 \
-m u32 --u32 "28 & 0xF8 = 0" \
-m string --algo bm --from 40 \
--u32 part says "grab 4 bytes starting from the 28th (0-indexed) byte of the packet... that is, bytes 28, 29, 30, 31, and mask the 32-bit value with 0x000000F8 ... that is only look at the high five bits of byte 31 ... and if it's 0, match." The
--from part, as mentioned, starts the string-search in the "question" section of the packet. This should avoid false positives.
Tested on RHEL 6.5 iptables 1.4.7.
To be pedantic, make another rule which tests the IP header field byte 0 for the header length and and matches the DNS query starting at byte 32.