So, is there a unix variant or unix inspired platform, (or a compiler if it is mostly up to the compiler?) which is standard but intentionally weird to produce bugs and point out undefined behavior and unportability?
Is it even a reasonable thing to try and implement?
...not if reasonable means "a tool anybody would reasonably want to use". But yes in the sense that this is what a normal C compiler is already. They inevitably do allow you to indulge in undefined behavior in a defined (predicatable) way. For example:
i = i++ + ++i;
Is undefined because you've modified
i multiple times between sequence points. However, almost certainly a given compiler will produce the exact same result every time. Of course, this won't be documented because you should not do it in the first place, but the compiler may know and can tell you if you ask it.
Put that in a file and compile it,
gcc undefined.c. No problems. Now try it with some warnings enabled,
gcc -Wall -pendantic undefined.c:
undefined.c: In function ‘main’:
undefined.c:8:4: warning: operation on ‘i’ may be undefined [-Wsequence-point]
i = i++ + ++i;
There you go. Now, keeping in mind that the compiler will produce the same result here everytime, you could decide to exploit this for whatever purpose. This means it is impossible for a compiler to "intentionally...produce bugs" when you do something wrong, if we define a "bug" as something you didn't want to happen. It doesn't know what you want to happen and it cannot. Therefore, it can't intentionally produce a result you can't predict.
It obviously can, however, "point out undefined behaviour".
If you want something that can go further than the compiler in spotting mistakes, look into valgrind. There are other (possibly very very very expensive) things along the same line, but this is the only one I've used.
Taking something you've developed exclusively on a 64-bit machine and compiling and running it 32-bit can also expose a lot of facepalm -- but obviously that is not the ideal way to correct mistakes.
Finally: the defacto way to test software is to test it. You write specific tests as you go, and you run them over and over again: at the end of the day, whenever you've added a new piece, etc. There is no substitute for this.