We have an java application that log4j2 to generate log file and have script to stop the process before restart it with another script. There is 5 minutes pause between stop and restart at the midnight daily. In the startup script, we use "mv" command to rename the log file with timstamp as extension. The issue is that one of the log file contains null characters( a few MB) at the beginning of the file and the log file becomes binary. a few observation notes to provide more context for this issue: 1.- Same startup script was used in other hosts with identical version of the java application. Don't have this issue at all. 2.- Occured occasionally. Ie., one week all 5 log files get corrupted and another week the log files are fine. 3.- Can not reproduce in similar developer Linux host; only in production Linux host. 4.- Size of the log file typically is about 4 - 6 GB daily. 5.- The application will be stop + 5 minutes pause + start by script at midnight daily. 6.- Used the hexdump to peek into the content of the binary log file. It had a few MB of null characters at the beginning and then followed by the normal typical ASCII content.
any suggestions will be appreciated. Thanks!