1

I'm using the following script to backup, using Borg Backup. But I'm running into an odd problem

Part of the script below does not execute. The script follows, then the output

SCRIPT:

#!/bin/bash                                                                                                                                                  

set -ex
set -o pipefail
export SERVER="myserver"
CLEVEL=zlib,9
. /home/faheem/.keychain/${HOSTNAME}-sh

#borg init -e none faheem@$SERVER:/mnt/backup-test                                                                                                         
borg create -c 30 --compression $CLEVEL --stats faheem@ramnode:/mnt/backup-test::`hostname`-`date +%Y-%m-%d:%H.%M`  /home/faheem/test-borg                   
{ borg check faheem@$SERVER:/mnt/backup-test 2>&1 1>&3 | tr '\r' '\n' | grep -Ev "^Remote:\s*(Checking segments.*)?$" 1>&2; } 3>&1
mapfile -t testarchives < <(borg list --short faheem@$SERVER:/mnt/backup-test)
borg extract -n "faheem@$SERVER:/mnt/backup-test"::"${testarchives[-1]}"

OUTPUT:

+ set -o pipefail
+ export SERVER=ramnode
+ SERVER=ramnode
+ CLEVEL=zlib,9
+ . /home/faheem/.keychain/orwell-sh
++ SSH_AUTH_SOCK=/tmp/ssh-F7Uzg6CeQoTY/agent.5660
++ export SSH_AUTH_SOCK
++ SSH_AGENT_PID=5661
++ export SSH_AGENT_PID
++ hostname
++ date +%Y-%m-%d:%H.%M
+ borg create -c 30 --compression zlib,9 --stats faheem@ramnode:/mnt/backup-test::orwell-2016-09-16:05.04 /home/faheem/test-borg
+ borg check faheem@ramnode:/mnt/backup-test
+ tr '\r' '\n'
+ grep -Ev '^Remote:\s*(Checking segments.*)?$'

So the last two lines don't execute. But if I comment out the earlier lines, those two lines execute. Does anyone know what is going wrong?

UPDATE: If I take out grep (i.e. replace)

{ borg check faheem@$SERVER:/mnt/backup-test 2>&1 1>&3 | tr '\r' '\n' | grep -Ev "^Remote:\s*(Checking segments.*)?$" 1>&2; } 3>&1

with

{ borg check faheem@$SERVER:/mnt/backup-test 2>&1 1>&3 | tr '\r' '\n' 1>&2; } 3>&1

I get

+ set -o pipefail
+ export SERVER=ramnode
+ SERVER=ramnode
+ CLEVEL=zlib,9
+ . /home/faheem/.keychain/orwell-sh
++ SSH_AUTH_SOCK=/tmp/ssh-F7Uzg6CeQoTY/agent.5660
++ export SSH_AUTH_SOCK
++ SSH_AGENT_PID=5661
++ export SSH_AGENT_PID
++ hostname
++ date +%Y-%m-%d:%H.%M
+ borg create -c 30 --compression zlib,9 --stats faheem@ramnode:/mnt/backup-test::orwell-2016-09-17:03.18 /home/faheem/test-borg
+ borg check faheem@ramnode:/mnt/backup-test
+ tr '\r' '\n'
Remote: Checking segments 0.0%
Remote:                         
+ mapfile -t testarchives
++ borg list --short faheem@ramnode:/mnt/backup-test
+ borg extract -n faheem@ramnode:/mnt/backup-test::orwell-2016-09-17:03.18

Does the Remote: line correspond to grep failing to match the pattern and returning non-zero?

Here are a couple of relevant questions:

Can grep return true/false or are there alternative methods

and

Avoid grep returning error when input doesn't match

The latter question appears to be a similar situation, if not the same.

3

The set -e means that if grep returns non-zero (ie the pattern is not matched) then that, combined with pipefail means that the pipe fails and causes the script to abort.

So either

  • remove set -e
  • remove pipefail
  • Add an || true in the grep segment to ensure that always returns zero
|improve this answer|||||
  • 1
    The truthfulness of a pipeline without pipefail is the exit status of the last command. Therefore, removing pipefail doesn't help him here if the grep is not matching. – jordanm Sep 16 '16 at 1:43
  • Right, but if other components of the pipeline are returning non-zero then it might. – Stephen Harris Sep 16 '16 at 2:08
  • Hi, Stephen. The third option looks like it would be the best. I don't need grep to succeed. So, if it doesn't match a pattern, that's considered an error? Weird. What would the syntax look like, exactly? The syntax of that particular command is already quite convoluted. – Faheem Mitha Sep 16 '16 at 9:57
  • I would try replacing grep -Ev "^Remote:\s*(Checking segments.*)?$" 1>&2 with {grep -Ev "^Remote:\s*(Checking segments.*)?$" 1>&2 || true;} but it might need some tuning. – Stephen Harris Sep 16 '16 at 10:54
  • 1
    Or you could use sed, since grep is designed to have its exit status set according to whether the pattern is matched or not. – Wildcard Sep 16 '16 at 22:45
1

grep is designed to set an exit status according to whether or not a particular pattern is found, and in some cases is even used only for this purpose and any output is suppressed. (That's the purpose of the -q switch to grep.)

sed is the "Stream EDitor" and is designed to filter a stream of text, converting it in some way (usually line by line) as it passes through.

Although there is a great deal of overlap between these two utilities, they do each have a specific purpose. In this case you are evidently not interested in whether or not the pattern is found, you simply want to filter the text stream before passing it along. That is a proper use case for sed, not for grep.


sed -E '/^Remote:\s*(Checking segments.*)?$/d' delme.txt

This is the exact equivalent of:

grep -Ev '^Remote:\s*(Checking segments.*)?$' delme.txt

...except that the grep command will give an error exit status if there are no lines besides those matching the pattern, and sed will not.

|improve this answer|||||
  • Thanks for the answer. On the face of it, sed does indeed look like a better choice here. Does the $/d simply remove a line that matches the regex on the left; i.e. ^Remote:\s*(Checking segments.*)?$? I'm not familiar with sed. – Faheem Mitha Sep 17 '16 at 1:00
  • @FaheemMitha, yes; lines which match the regex (/.../) have the delete command applied to them. Other lines have sed's default action applied, which is to print the line (unless suppressed with -n). – Wildcard Sep 17 '16 at 1:20
  • Ah. Thanks for the clarification, Wildcard. – Faheem Mitha Sep 17 '16 at 1:21
  • @FaheemMitha you're welcome. The best free tutorial for learning sed is the Grymoire page. I encourage you to read up on it--I've found sed indispensable as a sysadmin. :) – Wildcard Sep 17 '16 at 1:22
  • Thanks for the link, Wildcard. That looks like basically a book. – Faheem Mitha Sep 17 '16 at 1:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.