What is the best way to direct stdout & stderr to multiple logfiles?

Question

I have a purge process that I am running and the command I'm using is :

find $sentPurgerFolder -mtime +7 -print -delete >> $sentPurgeLogFile 2>&1

This code is in a while loop that is looping thru clients that have multiple folders of data to purge. The intent is to send all the purge info, regardless of the number of folders for that client to one log-file for that client. And that part seems to work pretty good.

My disconnect is I would like to send the same output to a master logfile, however the examples of 'tee' that I have seen give me pause. I do not know how I would integrate that command into my code here without doubling up the log data.

Can anyone lend some insight or make a suggestion?

Answer 1

Pipe each find ... 2>&1 to tee -a "$sentPurgeLogFile". This would append the output of the find command to the file given by the $sentPurgeLogFile value, but would also send the same output to the standard output of the while loop.

Redirect the output of the while loop to the "master" log file:

while ...; do

    # find commands here, e.g.
    find "$sentPurgerFolder" -mtime +7 -print -delete 2>&1 | tee -a "$sentPurgeLogFile"

done >master.log

This way, you would store the output for each individual client in its own logfile, but also store all output in master.log.

It is unclear whether the while loop runs a single find or multiple ones. If it runs a single find, remove the -a option of tee to create a new logfile for that client each time that command is run. If you run multiple find commands, you may want to only do that for the first tee. That's unless you want to store the accumulated logs over all invocations of the script, obviously (in which case you may also want to use >>master.log after the loop instead of >master.log).

Answer 2

OK I'll bite. Given what I imagine you're attempting to accomplish. Here's what I might attempt to use.

My assumption:

• dump results (append) to a master log
• dirs might have SUBdirs

#!/bin/sh -

masterlog="/some/place/some.file"

dirlist="
/one/place
/two/place
/three/place
"
for d in $dirlist
do
    cd $d
for f in `find . -mtime +7 -maxdepth 1 -type f`
do
    cat $f >>$masterlog
    echo ''>$f
done
done

exit

UNTESTED

What I imagined you were attempting to do was to collect the contents of all the individual logs into $masterlog, then empty the contents of those individual logs.

If I was correct. Then this'll git'er done. :)

HTH

Answer 3

I have found another method which totally negates this post but I will try to illustrate what I did....

So I have a client config file delimited by '%'. This is read, line by line by a sftp process that will push and pull data to/from the clients. It looks something like this:

...
client_1%FileContent_1%xml%...(client connection info)
client_1%FileContent_2%txt%...(client connection info)
client_1%FileContent_3%pdf%...(client connection info)
client_2%FileContent_1%xml%...(client connection info)
client_2%FileContent_2%pdf%...(client connection info)
...

So my purge process reads this and looks at the first three fields. They are used to assemble base-path for each client.

/xmit/client/outbound

Now the above folders each have a 'sent' or 'received' and 'log' folders.

/xmit/client/outbound/sent
/xmit/client/outbound/log

And this sftp process sends and receives files for these customers and when each file is processed, it puts said file in the /sent & /received folders and a full trace on the send/receive of said file go into the /log folders.

So the purge process that I am writing reads the same config file to locate the folders to purge from but i want to consolidate the log files of the purge for each client.

#
# Inits....
#
basepath=/xmit
workingdir=$basepath/bin/Purge
purgeProcessLogFile=$workingdir/outboundPurge.log
#
cnfgFl=/xmit/bin/OutBoundScripts/clientsftpconfigoutbound.txt
# Config File Ref
#
declare -i cnfgFlNbrOfLns=`wc -l "$cnfgFl" | cut -d ' ' -f 1`
# Count lines in config file...
#
# Set beginning line and work from beginning line to ending line...
# 
firstpass=1
declare -i cnfgFlStrtLn="53"
declare -i currentLnNbr=$cnfgFlStrtLn
lastClient='0'
#
#  Begin while loop...
#
while [ "$currentLnNbr" -le "$cnfgFlNbrOfLns" ]
do
  # 
  #  Set what line to read from and read the data in...
  #
  option="$currentLnNbr"p
  cnfgFlLn=`sed -n $option $cnfgFl`
  #
  # Increment counter for the next loop, you've allredy got the data...
  #
  declare -i currentLnNbr=`expr $currentLnNbr + 1`
  #
  #  Check if very first character in config line is not a "#" and proceed.
  #
  fld=`echo $cnfgFlLn | cut -c1-1`
  #
  if [ "$fld" != \# ]
  then
    #
    #  Get file content... (ex: '7501' )
    #
    cntnt=`echo $cnfgFlLn | cut -d "%" -f 2`
    #
    #  Extract account from 
    #
    currClient=`echo $cnfgFlLn | cut -d "%" -f 1`
    #
    if [ $currClient != \# ] 
    then
      #
      #  Check if on new account. If new account, set 'lastClient' to current account 
      #  and reset log files...
      #
      if [ "$lastClient" != "$currClient" ]
      then
        #
        lastClient=$currClient
        #
        #  Set current date and time stamp...
        #
        dt=`date +%Y%m%d``date +%H%M%S`
        #
        #  Set sent purge log...
        #
        sentPurgeFolder=$basepath/$currClient/outbound/sent
        sentPurgeLogFile=$sentPurgeFolder/SentPurgeLogFile\_$dt.log
        #
        #  Set sent log purge log...
        #
        sentLogPurgeFolder=$basepath/$currClient/outbound/logs
        sentLogPurgeLogFile=$sentLogPurgeFolder/SentLogPurgeLogFile\_$dt.log
        #
        #  Prime new logfiles...
        #
        cat /dev/null > $sentPurgeLogFile
        cat /dev/null > $sentLogPurgeLogFile
      fi
      #
      #  See if there are any files to purge...
      #
      declare -i cntntFilesToPurge=`find $sentPurgeFolder -mtime +7 -print | wc -l`
      if [ "$cntntFilesToPurge" -gt "0" ]
      then
        #
        # 'cd' to sent folder, purge and upate logfile...
        #
        cd $sentPurgeFolder
        dt1=`date +%Y%m%d``date +%H%M%S`
        echo $dt1 - Purging "$cntntFilesToPurge" "'"$cntnt"'" data files that are 7 days or older from $sentPurgeFolder >> $purgeProcessLogFile
        find $sentPurgeFolder -mtime +7 -print -delete >> $sentPurgeLogFile 2>&1
      else
        dt1=`date +%Y%m%d``date +%H%M%S`
        echo $dt1" - No ""'"$cntnt"'" data files that are 7 days or older to purge from $sentPurgeFolder >> $purgeProcessLogFile
      fi
      #
      # See if there are any logs to purge...
      #
      declare -i logFilesToPurge=`find $sentLogPurgeFolder -mtime +7 -print | wc -l`
      if [ "$logFilesToPurge" -gt "0" ]
      then
        #
        # 'cd' to logs folder, purge and upate logfile...
        #
        cd $sentLogPurgeFolder
        dt1=`date +%Y%m%d``date +%H%M%S`
        echo $dt1 - Purging "'"$cntnt"'" 'purge log' data are 7 days or older from $sentPurgeFolder >> $purgeProcessLogFile
        find $sentLogPurgeFolder -mtime +1 -print -delete >> $sentLogPurgeLogFile 2>&1
      else
        dt1=`date +%Y%m%d``date +%H%M%S`
        echo $dt1" - No logfiles for ""'"$cntnt"'" data that are 7 days or older to purge from $sentPurgeFolder >> $purgeProcessLogFile
      fi
    fi
  fi
done
#
#  E N D   O F   F I L E
#

The section :

  declare -i cntntFilesToPurge=`find $sentPurgeFolder -mtime +7 -print | wc -l`
  if [ "$cntntFilesToPurge" -gt "0" ]
  then
    cd $sentPurgeFolder
    dt1=`date +%Y%m%d``date +%H%M%S`
    echo $dt1 - Purging "$cntntFilesToPurge" "'"$cntnt"'" data files that are 7 days or older from $sentPurgeFolder >> $purgeProcessLogFile
    find $sentPurgeFolder -mtime +7 -print -delete >> $sentPurgeLogFile 2>&1
  else
    dt1=`date +%Y%m%d``date +%H%M%S`
    echo $dt1" - No ""'"$cntnt"'" data files that are 7 days or older to purge from $sentPurgeFolder >> $purgeProcessLogFile
  fi
  #
  # 'cd' to logs folder, purge and upate logfile...
  #
  declare -i logFilesToPurge=`find $sentLogPurgeFolder -mtime +7 -print | wc -l`
  if [ "$logFilesToPurge" -gt "0" ]
  then
    cd $sentLogPurgeFolder
    dt1=`date +%Y%m%d``date +%H%M%S`
    echo $dt1 - Purging "'"$cntnt"'" 'purge log' data are 7 days or older from $sentPurgeFolder >> $purgeProcessLogFile
    find $sentLogPurgeFolder -mtime +1 -print -delete >> $sentLogPurgeLogFile 2>&1
  else
    dt1=`date +%Y%m%d``date +%H%M%S`
    echo $dt1" - No logfiles for ""'"$cntnt"'" data that are 7 days or older to purge from $sentPurgeFolder >> $purgeProcessLogFile
  fi

Was before:

  #
  # 'cd' to sent folder, purge and upate logfile...
  #
  cd $sentPurgeFolder
  dt1=`date +%Y%m%d``date +%H%M%S`
  find $sentPurgeFolder -mtime +7 -print -delete >> $sentPurgeLogFile 2>&1
  #
  # 'cd' to logs folder, purge and upate logfile...
  #
  cd $sentLogPurgeFolder
  dt1=`date +%Y%m%d``date +%H%M%S`
  find $sentLogPurgeFolder -mtime +1 -print -delete >> $sentLogPurgeLogFile 2>&1

This has satisfied my needs in terms of a 'purge' log.