As mentioned in Why does a software package run just fine even when it is being upgraded?, the lock is placed on inode not on filename. When you load and execute a binary, the the file is marked as busy - which is why you get ETXTBSY (file busy) error when you try to write to it.
Now, for shared libraries it is slightly different: the libraries get memory mapped into the process' address space with
MAP_DENYWRITE may be specified, at least Glibc on Linux silently ignores it (according to the man page, feel free to check the sources) - check this thread. Hence you actually are allowed to write the file and, as it is memory mapped, any changes are visible almost immediately - which means that if you try hard enough you can manage to brick your machine by overwriting the library.
The correct way to update therefore is:
removing the file, which removes reference to the data from the file system, so that it isn't accessible for any newly spawned applications that might want to use it, while keeping the data accessible for anyone who already has it open (or mapped);
creating a new file with updated contents.
Newly created processes will use the updated contents, running applications will access the old version. This is what any sane package management utility does. Note that it's not completely without any danger though - for example applications dynamically loading code (using
dlsym() and friends) will experience troubles if the library's API changes silently.
If you want to be on the really, really safe side, shut down the system, mount the file system from another operating system instance, update and bring up the updated system again.