This is what is called an integer overflow error. According to @doneal24's answer, Bash uses 64-bit integers, but I'll use an 8-bit integer in my explanation to make it easier to show what is going on.
An 8-bit integer stores a positive or negative number inside 8 bits, but not all of those bits store the value of the integer - the first bit of the number, the sign bit, denotes whether the number is positive or negative (with "0" meaning positive, and "1" meaning negative), with the next seven bits composing the value of the number in question using standard binary notation.
So, for instance, an integer with a value of one looks like this:
Then, we can begin running your program, and start increasing the value of that integer by multiplying it by the value needed to calculate the next factorial. So, if we multiply that by 2, and then begin repeating that process, we get the following:
0000 0001 x 0000 0010 = 0000 0010
0000 0010 x 0000 0011 = 0000 1010
0000 1010 x 0000 0100 = 0001 1000
0001 1000 x 0000 0101 = 0111 1000
Now we're getting close. The next iteration is where the error happens:
0111 1000 x 0000 0110 = 0010 1101 0000
You can notice that it's overflowed past the eight bits that were assigned to storing the value of the integer - that means that it's overwritten whatever happened to be stored in the memory location immediately before where the integer was stored, and that's bad news, since it can lead to corruption of the processes of the computer; indeed, there have been hackers who have used similar security holes to attack computers in the past.
However, you can also see that the eight bits stored in the location of the integer are "1101 0000". This means that, when your computer interprets its value, it will read the first bit as the sign bit, indicating a negative number, and then read the rest of the number as "101 0000", or 80, and thus it would display the value of this operation as "-80".