On most (but not all) unix systems, PAM handles authentication, so you would need to plug your script into the PAM infrastructure. You'll need to read some parts of the PAM administrators' guide.
The PAM configuration is in /etc/pam.conf or /etc/pam.d/*. You'll want to run your script in the auth stack. Jump over it when authentication succeeds (something like [success 1] on the last required component) and make it a requisite entry.
You don't specify what you want to base your number of login attempts on. Per user? Per unit of time? Depending on the answer, you may need to include additional PAM configuration or do more work in your script.
Turning off the machine is rather silly. What if it was doing something useful, like a background computation or a remote user? The rate of login attempts at the console isn't worrying anyway, unless someone has brought hardware to automate them, in which case they can grab your hard disk and run.