I have a directory that contains multiple directories each full of log files. Every log file is named in the format of YYYY-MM-DD.log. So it would be /path/to/logs/tokyo/2017-06-29.log, /path/to/logs/london/2017-06-29.log, etc.
A new log is opened every day at midnight with the new date, and the old ones will never be written to again.
There are literally thousands of these logs going back years, and while they're just text files they are starting to take up a non-negligible amount of space. So what I'm trying to do is write a script that looks in all of these log directories, finds any dated before today, and bzips them. Here is what I have so far:
#!/bin/sh LOGDATE=$(date +%Y-%m-%d) LOGPATH="/path/to/logs" for i in $(ls $LOGPATH/*/*.log); do if [ "$i" != "$LOGDATE.log" ];then tar cfjv $i.tar.bz2 $i rm $i else fi done
There are two problems I'm running into. First, the $(ls ...) in the for loop errors out saying the argument list is too long. Second, in the tar statement, I want the bzipped output file to be in the directory where the log file was, but it seems to instead drop them in the directory the script is being run from, and I can't find a way to make the loop use the correct directory for every matching file it encounters.
Presumably this would be better handled with a well written find and xargs command, but I've been unable to work one out that does what I need it to do.
The end goal of this script is to find all log files in the log directory structure, check that it is not today's log, bzip it in place, and delete the text log. It would be run once immediately to take care of all the old logs, and then added to crontab to run just after midnight every day.
What do I need to do to correct my script? Or, is there better way altogether?
The only real restriction is that it's a freebsd server so it needs to use either sh or csh, and use freebsd tools (meaning for example bsd's find instead of GNU find which has more extensive options).