Are there any better alternative to chroot environment? I'm thinking about running nginx on a jailed environment.
BTW, I'm on OpenVZ VPS, so modifying the Kernel is a no-no.
(I think that prevents me from installing SELinux, AppArmor, etc.)
Forget the myths about how Chroot isn't a security feature... it is integrated as a security feature into mainstream applications of all sizes and flavors for a reason
OpenVZ is itself basically Chroot on Steroids, and this should, or will be, all that you will need. as long as it is done conforming to the Chroot setup standards and typical procedure.
Generally, If you are simply trying to provide an additional layer of security for NGinx, then use Chroot. You are mainly trying to provide privacy between users by doing this, or providing security from poorly coded content management systems.
this helps protect, for a few minor examples, against directory traversal attacks, remote file inclusion bugs, or executing applications on the base system. it can also help protect against any known, unknown, or future NGinx vulnerabilities, by containing damage to the Chroot directory.
with this said, however, be sure to secure your base system, and check what version of NGinx your operating system repositories are using, and compare it against any security advisories that is currently available for NGinx. you may want to compile to latest stable build and use that instead.
there is a useful application by R-FX Networks that can help protect against some kinds of attacks against a running daemon, without using AppArmor, or SELinux. It is called SNIV.
for more information on why Chroot is still a totally valid security tool in today's day and and age; see this answer I recently gave for a more thorough explanation this scenario.