5

I'm trying to do the following within a for loop:

  • Find files that satisfy a condition
  • Echo the name of the files to a log file.
  • Gzip the file.

I can get the script to find the files and echo their names to the screen but cannot pipe them to a file. (I have not got as far testing the gzip part)

Here's the relevant portion of the script (The $LOGCLEAN_FILE exists and is written to in an earlier portion of the script):

for F in `find . -type f \( ! -name '*.gz' \) -a \( ! -name '*.Z' \) -mtime +7`
do

        {
        print "Will be compressing file  ${F}" >> $LOGCLEAN_FILE
        } ; 
        ##gzip $F
done

If I run the script without the " >> $LOGCLEAN_FILE" text then the output displays on the screen.

What am I doing wrong?

Criticisms are welcome - I'm still learning.

4
  • The command to write output in the shells is echo, not print.
    – vonbrand
    Mar 15, 2013 at 19:05
  • @vonbrand, it is print in the shell that morgon specified (ksh). Mar 15, 2013 at 20:44
  • 2
    What is the value of $LOGCLEAN_FILE? If it's a relative path, have you changed directories? Mar 15, 2013 at 22:13
  • 1
    Gilles - your comment actually pointed me to the problem with my script. The script changed directory after the setup of the $LOGCLEAN_FILE variable and wrote the rest of the output to a different directory instead of the intended location. If you can submit the comment as an answer I'll mark it as the answer.
    – morgan_g
    Mar 16, 2013 at 18:45

5 Answers 5

4

There really is no need for reproducing find's output ina shell loop. If you want to pack the list of filenames, the generic formula is:

find ... | gzip > logfile.gz

If you want to gzip the files themselves, it changes to:

find ... | tar -czvf archive.tar.gz -T -

which tells tar to read list of file names to work on from a file and the single - stands for standard input. (The -T AKA --files-from= option is present in GNU tar, I'm not sure about other flavours.) Of course this breaks if you manage to work on files which contain \n in their names.

4

You cannot post-process the output of find reliably. Use -exec in find:

find . -type f ! -name '*.gz' ! -name '*.Z' -mtime +7 -exec sh -c '
  for i do
    printf "%s\n" "Will be compressing file $i"
    gzip "$i"
  done' sh {} + >> log

With the GNU implementation of find, you can even get away without running sh:

find . -type f ! -name '*.gz' ! -name '*.Z' -mtime +7 \
  -printf 'Will be compressing file %p\n' -exec gzip {} + >> log
5
  • Good advice, but I'm puzzled: why isn't the script as posted working (assuming tame file names)? Mar 15, 2013 at 22:12
  • @Gilles, That I do not know either. It smells like a end of a long week user error, where you're not looking at the file you thought you were. Or it could be another process that truncates or recreates the same file in the mean time. Mar 15, 2013 at 22:19
  • I am looking at the correct files , because if I remove the statement to pipe output to a file, the list of files displayed on the screen is the desired list of files. My original post was not clear - I apologise.
    – morgan_g
    Mar 16, 2013 at 7:12
  • My problem with using the -exec option is that it changes the result of the previous find statement. If I run the find command as type in my post at a shell prompt , then it displays a correct list of files, If I add "-exec ls -al {} \;" then the list displayed is markedly different. I suspect however that I will be submitting a separate question about that.
    – morgan_g
    Mar 16, 2013 at 7:21
  • On what basis do you make the comment? You cannot post-process the output of find reliably. I use simliar commands to move files, mount file systems etc regularly in daily work.
    – morgan_g
    Mar 16, 2013 at 7:23
2

put your redirect (>>) outside the loop (after done) like so:

for i in $(find . -type f \( ! -name '*.gz' \) -a \( ! -name '*.Z' \) -mtime +7);
  do echo "Will be compressing file  ${i}";
  ##gzip $i;
done >> $LOGCLEAN_FILE

Or, if you plan on eventually uncommenting your gzip command you might consider:

for i in $(find . -type f \( ! -name '*.gz' \) -a \( ! -name '*.Z' \) -mtime +7);
  do gzip "$i" && echo "Successfully compressed ${i}";
done >> $LOGCLEAN_FILE

the && means only do the next command if the previous command exits with 0 (no error).

2
  • Will test once I am back at my desk. Thanks for the feedback
    – morgan_g
    Mar 16, 2013 at 6:53
  • This does not address piping output inside the for loop! Feb 1, 2015 at 2:14
1

As filenames can have both spaces and new-lines a probable approach would be:

#!/bin/bash

logclean_file="logclean_file.txt"

ts="$(date)"

printf "%s\n" "$ts" > "$logclean_file"

# Set IFS blank
# -r Backslash does not act as an escape character. The  backslash
#    is  considered  to be part of the line.  In particular, a
#    back-slash-newline pair may not be used as a line continuation.
# -d delim
#    The  first  character  of  delim is used to terminate the input
#    line, rather than newline.
# 
# Here setting -d to nul or 0x00. This enables us to capture any file-
# names with the print0 from find.
#
# fn The variable to read into.
#
while IFS= read -r -d $'\x00' fn; do
    printf "Will perhaps be compressing file %s\n" "$fn" >> "$logclean_file"
    # If file + gz does not exist 
    if [[ ! -e "$fn.gz" ]]; then
            if gzip command "$fn"; then
                    echo "Horray! success!" >> "$logclean_file"
            else
                    echo "Harf! Gzip failed." >> "$logclean_file"
            fi
    else
            echo "Nah. Already exists." >> "$logclean_file"
    fi
done < <(find . -type f \( ! -name '*.gz' \) -a \( ! -name '*.Z' \) -print0)
# Notice -print0 at end which means find will print filenames, - separating
# them with 0x00 instead of new-line

About portability:

Neither echo nor print are really portable. While print is unique to ksh93, echo only got standardized very late in the POSIX process, and older versions cannot be relied on to produce predictable results. See echo vs print and Why is printf better ....

printf "Some %s\n" "$var" >> "$foo"
# Or
echo "Some $var" >> "$foo"

About Style:

  • quote variables.
  • Consider using lowercase of user variables (your own variables).

Some ref:

For the TLDP guides, and if you are using stylish, I'd recommend one of these for screen-reading.

5
  • Nobody mentioned bash. print is a ksh (and zsh) command. Mar 15, 2013 at 20:33
  • @StephaneChazelas: Oh, dang. Sometimes I seem to have extremely selective eyesight. How did I miss that? - Well, Thank you for correcting me, - I'll delete :)
    – Runium
    Mar 16, 2013 at 7:03
  • @Sukminder - thanks for Greycat's links on shell quotes. It's a good resource. Mar 17, 2013 at 13:06
  • Thanks to all for the advice will check out sukminder's link references
    – morgan_g
    Mar 18, 2013 at 11:39
  • @HenkLangeveld: Thanks for edit! I won't delete after all then.
    – Runium
    Mar 20, 2013 at 0:39
0

If you are trying to pipe to an output file, iterating over file paths...

Use find's exec capabilities

... to process your iterations.

I wanted to iterate over an explicit files list via subshell, and call expand which outputs to stdout, and overwrite each individual file. A problem addressed by this answer.

find $(git ls-files | grep '.py$') \
    -exec bash -c 'expand -t 4 "$0" > /tmp/e && mv /tmp/e "$0"' {} \;

I'm reposing a slightly modified version of that answer since the other answers in this thread don't provide a general purpose solution. This is the only thing I've seen that works, which only applies when iterating over file paths.

I can imagine a solution that abusively creates files named after your iteration items so that you can iterate over that name using find.


Failed attempt:

So like the find exec call, I tried using bash -c ... inside the for loop. But I ran into the same problem of empty output files.

Disclaimer: I'm not sure what my problem was exactly, in all attempts prior to using find exec, all the output files I expected to be written to, got overwritten, my problem was that they were empty files afterward.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .