0

I need to make a list of files bigger than 10G that will be the ignorefile input for a duply backup. However, I have some files which I don't have permissions to read. I want to ignore those that will be handled later and make the list clean with out the 'Permission denied' message.

I can make the list like this:

$ find data/ -size +10G
/data/bigfile1
find: `/data/badfile1': Permission denied
find: `/data/badfile2': Permission denied
/data/bigfile2

Then I need to remove the lines that contain the 'Permission denied' or 'find:' strings. I'm not sure if the order of the list output is relevant, but that's what I'm seeing. I'm trying several options but can not manage to remove the lines. I'm following this question, that seems similar but I don't understand why it doesn't work. https://stackoverflow.com/questions/3548453/negative-matching-using-grep-match-lines-that-do-not-contain-foo

I'm trying to ways, first with grep:

$ find data/ -size +10G | grep -v 'find:'
find: `/data/badfile1': Permission denied
/data/bigfile1
find: `/data/badfile2': Permission denied
/data/bigfile2

Also using awk:

$ find data/ -size +10G | awk '!/find:/'
find: `/data/badfile1': Permission denied
find: `/data/badfile2': Permission denied
/data/bigfile1
/data/bigfile2

In any case the positive selection works fine. So I don't know what to do.

  • 2
    find [....] 2> /dev/null. What you're seeing is standard error output, which is not sent into the pipeline, so grep will not see it, nor will any other things in a pipeline. You can see this by running your find command and sending the output into a new file; you will not see the errors in the new file. – DopeGhoti Aug 8 '17 at 16:24
0

IO Redirection

The problem is not with your script, but is a feature of UNIX IO redirection.

Consider, the following command

find data/ -size +10G | grep -v 'find:'

only the stdout stream is being sent to the piped grep command, the error messages are in a seperate output stream, stderr.

So, even though you are seeing the error messages in your terminal session, due to the fact that the terminal is printing both stdout and stderr to the console, your grep command is not receiving those lines of output. This is due to how the | redirects output. The pipe | only connects stdout to the next command's stdin so the stderr does not go into the next command.

Most *NIX utilities function this way, sending errors to the stderr output stream so that in situations like you are encountering there is no need to filter these messages out.

Testing

To see what the exact stdout of the above command will be when it is passed to grep do the following:

find data/ -size +10G > out.txt

You should still see the errors in your console, but the stdout output stream will have gone to a new file, out.txt. Look in there and you will not see the lines of error messages.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.