If you have nosharedscripts set (the default), and the prerotate script exits with an error the affected log file will not have any further action taken on it*.
So, in theory, you could have something like (warning, untested):
/var/log/application.log {
nosharedscripts
prerotate
logfile=$1
lsof $logfile >/dev/null
exit $?
endscript
...
}
so if lsof can't find any processes with $logfile open, the prerotate script will exit with 1, and logrotate will not take any action on that log.
*From the logrotate(8) man page on linux.die.net:
nosharedscripts
Run prerotate and postrotate scripts for every log file which is rotated
(this is the default, and overrides the sharedscripts option). The absolute
path to the log file is passed as first argument to the script. If the
scripts exit with error, the remaining actions will not be executed for the
affected log only.
...
prerotate/endscript
The lines between prerotate and endscript (both of which must appear on
lines by themselves) are executed (using /bin/sh) before the log file is
rotated and only if the log will actually be rotated. These directives may
only appear inside a log file definition. Normally, the absolute path to
the log file is passed as first argument to the script. If sharedscripts is
specified, whole pattern is passed to the script. See also postrotate. See
sharedscripts and nosharedscripts for error handling.
This may be dependant on your logrotate version, but I can't find any documentation on which versions do / do not behave this way.