"Site-specific" data in FHS and "site packages" in Python are two different things.
FHS: site-specific data
As it states in documentation:
/srv : Data for services provided by this system
This main purpose of specifying this is so that users may find the location of the data files for particular service, and so that services which require a single tree for readonly data, writable data and scripts (such as cgi scripts) can be reasonably placed. Data that is only of interest to a specific user should go in that users' home directory.
The methodology used to name subdirectories of /srv is unspecified as there is currently no consensus on how this should be done. One method for structuring data under /srv is by protocol, eg. ftp, rsync, www, and cvs. On large systems it can be useful to structure /srv by administrative context, such as /srv/physics/www, /srv/compsci/cvs, etc. This setup will differ from host to host. Therefore, no program should rely on a specific subdirectory structure of /srv existing or data necessarily being stored in /srv. However /srv should always exist on FHS compliant systems and should be used as the default location for such data.
So here you can put data files for a particular service, such as FTP, WWW, or CVS (in contrast data that only pertains to a specific user placed in the /home/ directory).
Python: site packages
Here it's just a location where 3rd party Python packages (not part of core Python distribution) are installed. E.g. for Python3 in Ubuntu it's:
>>> import site; site.getsitepackages();
['/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.4/dist-packages', '/usr/lib/dist-python']
Update: meaning of 'site' in FHS
By 'site' it means resource that's served to users on different hosts via network. E.g. 'web-site' is a set of web-pages served via http, or if you want to give other users access to your git-repository and running git daemon so they can connect to it via git/ssh protocols.