Linux only has cooperative locks. Nothing will prevent a process from opening a file that it has permission to open. You would need to call a locking mechanism in all processes using the file to prevent concurrent access. However, what you're trying to do doesn't have anything to do with concurrent access, as far as I can see.
If you remove a file that is open in a process, the file isn't actually removed: only its name is removed. The file itself is actually removed only when it no longer has any name (zero hard link) and it is no longer open. So if you run rm server.log while the JBoss application still has it open, that won't free any disk space until the JBoss application is restarted (or closes the log file for some other reason).
If you run > server.log or echo "" > server.log, that does truncate the file to 0 or 1 bytes respectively, so the disk space is freed. If the JBoss application was writing to the file, it will keep writing at the same position, 500+MB into the file. This results in a sparse file, where the file contains null bytes up to the position where the application started writing to it. For example, if the file was 500MB when you truncated it, and the application has kept writing 1MB, then the file size is 501MB, but the disk space occupied by the file is only 1MB, and the first 500MB of the file are null bytes.
If you want the application to write at the beginning of the file, you'll have to tell it to do that. Typically, you'd tell it to close its current log file and start a new one. As grebneke commented: see How to clear server.log in JBoss?
