I have several linux systems running various homemade applications, some web apis, some background data crunching, multiple databases and things I haven't even found yet. These systems were set up by multiple people over several months, and no one really knows how each server is used. Some of the people setting up programs on these were interns or contractors who have since departed. So we don't know how anything is configured.
I'm trying to figure out what log files are being written, especially those being written in weird places (i.e. not under /var/log). I'm also trying to find all log files not being rotated, especially if they are growing rapidly. I found two servers in the past week running up against their disk limit and crashing processes. My goal is to reconfigure each application to do something sane for logging, and eventually send it all to an ELK stack, but for now I just need to figure out what I've got.
So to start I'm trying to find everything that's a log file. That's hard if they're scattered randomly in the system. Some under /home/someuser, one was under /root, some in /tmp and one in /var/lib.
My first thought to find log files was to find any file modified recently. See this answer: https://askubuntu.com/a/704163/139584
This gets me a lot of noise though. Databases persist things to disk, so they write files, system updates replace binaries so those are modified, and users have modified stuff in their homes.
My next thought was to find by name. Most log files end with .log, but some do not. Maybe some have "log" somewhere in the pathname. See this answer: https://askubuntu.com/a/144703/139584
Once I have a list of logs, I can scan the logrotate rules to find anything matching. That should be easy enough with for and grep.
Does anyone have a better idea of how to enumerate stray log files in an undocumented linux system?