With very few exceptions, if somebody has your hardware in their hands, they can duplicate everything, simply by accessing and copying the whole storage.
There's no extra encryption that would help. If you encrypt the disk, the disk encryption key has to be readable somewhere. Disk encryption is useless in your scenario.
There is hardware that can't be easily duplicated, such as smartcards. However, even if you connect a smartcard reader to the RPi, the thief can just steal the card with the Pi.
You can protect against someone stealing (or borrowing) the Pi or the SD card if the SD card is encrypted and the key is not available to the thief. This means that you or someone you trust would have to type in a password, or insert an SD card that contains the encryption key for a USB key, in order for the Pi to boot. This isn't perfect protection: someone could make a dump of the RAM — but since the RAM is soldered on a Pi, that's a relatively difficult hardware attacks. If you're on an RPi budget, you probably don't need that level of resistance.
There are hardware platforms with an integrated tamper-resistant key storage: TPM on PC platforms, ARM system-on-chips with TrustZone and hardware root of trust (TrustZone alone as a CPU feature isn't enough). The hardware cost is an order of magnitude more than a Raspberry Pi. Note that even these systems wouldn't prevent theft; they would only prevent the thief from duplicating the client device.
Another route to protection is physical protection: put the device in a locked box that is securely fastened to a building fixture.
If you can't prevent adversaries from physically accessing the client device, then you can't prevent them from stealing your keys. All you can do is attempt to detect the theft. For example, if multiple clients turn up with the same client certificate, there's definitely something wrong (but you face a hard decision if you can't tell which one is the legitimate one: allow access to all, or deny access to the legitimate one).