The reason why TCP/IP sockets use file descriptors is that, when the sockets interface was first designed and implemented (in BSD Unix, in 1983), its designers felt that a network connection was analogous to a file - you can
close both, and that it would fit well with the Unix idea of "everything is a file".
Other TCP/IP network stack implementations didn't necessarily integrate with their OS's file-I/O subsystem, an example being MacTCP. But because the BSD sockets interface was so popular, even these other implementations chose to replicate the socket API with its Unix-like functions, so you got "file descriptors", only used for TCP/IP communication, on systems that didn't otherwise have file descriptors.
The other part of your question is why is there a limit? It's because the quickest way to implement a file descriptor lookup table is with an array. Historically, the limit was hard-coded into the kernel.
Here's the code in Unix release 7 (1979) with a hard-coded limit 20 file descriptors per process:
By comparison, Linux dynamically allocates space for a process's file descriptor table. The absolute limit defaults to 8192, but you can set this to whatever you like. My system lists 191072 in
Despite there being no absolute limit in Linux any more, nonetheless we don't want to let programs go crazy, so the administrator (or the distribution packager) generally sets resource limits. Take a look at
/etc/security/limits.conf, or run