1

I'm using rsync to backup some files:

rsync -PaSh --stats --delete -e "-i key -p XXXX" "/source/" username@192.168.0.1:/destination/ 2> output.txt | grep -e 'bytes  received' -e 'total bytes' -e files -e 'total file size:' >> output.txt

because there are thousands of files, I only want to see any errors and a summary at the end.

The above command outputs this:

rsync: delete_file: unlink(test/test.txt) failed: Permission denied (13)
Number of files: 12 (reg: 10, dir: 2)
Number of created files: 0
Number of regular files transferred: 0
sent 382 bytes  received 137 bytes  41.52 bytes/sec

I want to convert to uppercase any errors only (to draw attention to them) and leave the summary unchanged.

So it would look like this:

RSYNC: DELETE_FILE: UNLINK(TEST/TEST.TXT) FAILED: PERMISSION DENIED (13)
Number of files: 12 (reg: 10, dir: 2)
Number of created files: 0
Number of regular files transferred: 0
sent 382 bytes  received 137 bytes  41.52 bytes/sec

How can I achieve this?

Thanks

  • Side comment/question not mentioned in my answer: wouldn't it be easier to just tail the rsync stdout to get the summary info? Or does it get displayed elsewhere besides the very end of the rsync run? (If it would work that's a lot better than scanning the whole thing with grep.) – B Layer Jan 6 '18 at 20:36
3

You can effectively swap stdin and stdout with the help of a temporary file descriptor (fd 3 here):

cmd 3>&2 2>&1 1>&3- | tr '[:lower:]' '[:upper]'

That says "point new fd at the place stderr points to, point stderr where stdout points, point stdout to where stderr originally pointed"...easy, right? :)

Or in other words: now the stderr from cmd will go through the pipe like stdout while the original stdout has been redirected to stderr. And the tr command will do the upper-case conversion.

Update: Though you accepted the answer above your wishlist included the ability to manipulate stdout and stderr simultaneously. So let's try to tackle that...keeping in mind that we're bumping up against some things I don't do too often!

A simple swapping of stdout/stderr won't do anything useful. With or without the swap you can't use two different streams through a single pipeline. The first thing that comes to mind as an alternative is to use FIFOs (named pipes):

mkfifo /tmp/fifo1 /tmp/fifo2

# run the key command in the background, in a subshell to suppress
# job control messages (e.g. "[1] 12345" and "[1]+ Exit ...")
( cmd > /tmp/fifo1 2> /tmp/fifo2 & )

# concatenate tr/stderr and grep/stdout using process substitution
# (I'm assuming the summary information occurs at the end of rsync output)
cat <(cat /tmp/fifo2 | tr '[:lower:]' '[:upper:]') <(grep ... /tmp/fifo1) 

rm -f /tmp/fifo1 /tmp/fifo2

You could actually put this all on one line if so inclined!

mkfifo /tmp/fifo1 /tmp/fifo2; (cmd > /tmp/fifo1 2> /tmp/fifo2 &); cat <(cat /tmp/fifo2 | tr '[:lower:]' '[:upper:]') <(grep ... /tmp/fifo1); rm -f /tmp/fifo1 /tmp/fifo2

My sanity checks are working okay so give it a try.

  • Thanks for your reply; I'm getting closer. I'm now using this: rsync -PaSh --stats --delete -e "-i key -p XXXX" "/source/" username@192.168.0.1:/destination/ 3>&2 2>&1 1>&3 | tr [:lower:] [:upper:] > output.txt | grep -e 'bytes received' -e 'total bytes' -e files -e 'total file size:' >> output.txt which works in converting stderr to uppercase. Unfortunately I lose the ability to append stdout to the file and instead it just appears on screen. Do I need to add 1> somewhere? – NoExpert Jan 6 '18 at 16:44
  • @NoExpert You want to save to file the original stdout straight from the rsync command, yes? – B Layer Jan 6 '18 at 17:02
  • Only one stream can go through those pipes. You can't have both the tr for the stderr and grep for the stdout in a single command. – B Layer Jan 6 '18 at 17:07
  • Yes, after greping what I need, so as in my example in the original above. Thank you. EDIT: OK I've seen your updated answer now. That makes sense. I guess I'll have to live with it. – NoExpert Jan 6 '18 at 17:08
  • I think you can accomplish what you want if you're willing to use named pipes (FIFOs). I'll post an update to my answer. – B Layer Jan 6 '18 at 20:06
1

To convert stderr to uppercase without changing where stdout and stderr go to:

{ rsync ... 2>&1 >&3 3>&- | perl -Mopen=locale -pe '$_=uc' >&2 3>&-; } 3>&1

That is duplicate the original stdout onto fd 3, so we can restore it for rsync.

Here using perl instead of tr as it gives better results for things like Stéphane -> STÉPHANE (STéPHANE with GNU tr) or traffic -> TRAFFIC (TRAffiC with GNU tr/sed/awk).

Note that now that stderr goes through a pipe, that can affect the relative order of stdout and stderr messages as the stderr messages are now being delayed.

See

for some other approaches that you might find useful.

0

Use below command instead of your command. I have added one extra command after redirecting the error to output.txt

I have added sed -i "s/.*/\U&/g" output.txt
As tested its works fine


rsync -PaSh --stats --delete -e "-i key -p XXXX" "/source/" username@192.168.0.1:/destination/ 2> output.txt ;sed -i  "s/.*/\U&/g" output.txt| grep -e 'bytes  received' -e 'total bytes' -e files -e 'total file size:' >> output.txt  

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.