I think I would attack this issue in a couple of ways. For starters I would try and diagnose it myself using the following methodologies.
1. obnam logs
For starters you can log messages from
obnam like so:
$ obnam --log obnam.log
You can increase the logging level via the
--log-level switch as well to get more details.
--log=FILE write log entries to FILE (default is to not write log
files at all); use "syslog" to log to system log, or
"none" to disable logging
--log-level=LEVEL log at LEVEL, one of debug, info, warning, error,
critical, fatal (default: info)
You can also get a profile of what
obnam is doing as follows from this excerpt in the project's FAQs:
OBNAM_PROFILE environment variable contains a file name, the
profiling data gets stored there, and can be viewed later with
$ OBNAM_PROFILE=obnam.prof obnam ... obnam-viewprof obnam.prof | less
Performance issues that are not related to a particular setup can also
be observed using the
3. Open a ticket
If the performance is still not determined doing some self driven investigation then I would open a ticket up on the project's website. From what I was able to gather the developer(s) are somewhat responsive and they would likely be the best at rooting out issues with their project.
obnam appears to only use SFTP so it should be pretty obvious what's causing the problem. I would also consider baselining the SFTP performance by itself so you can see what the theoretical maximum should be with your system + network connection prior to trying to get this info out of
obnam tests themselves.
Additional data points
#1 - blog post comparing obnam vs. rsnapshot
Found this blog post where the author did a comparison of several options in this category. The article is titled: Comparing rsnapshot and obnam for scheduled large backups.
The article highlighted some very poor performance, IMO, with
obnam that would seem to jive with what you're describing.
After backing up /home completely (which took several days!), a new run, several days later took (timing by the Linux time command):
Backed up 3443706 files, uploaded 94.0 GiB in 127h48m49s at 214.2 KiB/s average speed830 files; 1.24 GiB (0 B/s) real 7668m56.628s user 4767m16.132s sys 162m48.739s
From the obname log file:
2012-11-17 12:41:34 INFO VFS: baseurl=/home read=0 written=0
2012-11-21 23:09:36 INFO VFS: baseurl=/backups/backup_home read=2727031576964 written=150015706142
2012-11-21 23:09:36 INFO Backup performance statistics:
2012-11-21 23:09:36 INFO * files found: 3443706
2012-11-21 23:09:36 INFO * uploaded data: 100915247663 bytes (93.9846482715 GiB) 2012-11-21 23:09:36 INFO * duration: 460128.627629s
2012-11-21 23:09:36 INFO * average speed: 214.179341663 KiB/s
2012-11-21 23:09:36 INFO Backup finished. 2012-11-21 23:09:36 INFO Obnam ends
2012-11-21 23:09:36 INFO obnam version 1.2 ends normally
So: ~5 days for backing up ~100 GB of changed data… Load was not high
on the machines, neither in terms of CPU, nor in terms of RAM. Disk
usage in /backups/backup_home was 5.7T, disk usage of /home was 6.6T,
so there is some dedup, it seems.
A full backup of /home to (according to the log file):
[27/Nov/2012:12:55:31] /usr/bin/rsnapshot daily: started
[27/Nov/2012:12:55:31] echo 17632 > /var/run/rsnapshot.pid
[27/Nov/2012:12:55:31] mkdir -m 0700 -p /backups/backup_home_rsnapshot/
[27/Nov/2012:12:55:31] mkdir -m 0755 -p /backups/backup_home_rsnapshot/daily.0/
[27/Nov/2012:12:55:31] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /home /backups/backup_home_rsnapshot/daily.0/localhost/
[28/Nov/2012:23:16:16] touch /backups/backup_home_rsnapshot/daily.0/
[28/Nov/2012:23:16:16] rm -f /var/run/rsnapshot.pid
[28/Nov/2012:23:16:16] /usr/bin/rsnapshot daily: completed successfully
So: ~1.5 days for a full backup of 6.3TB. An incremental backup a
day later took:
[29/Nov/2012:13:10:21] /usr/bin/rsnapshot daily: started
[29/Nov/2012:13:10:21] echo 20359 > /var/run/rsnapshot.pid
[29/Nov/2012:13:10:21] mv /backups/backup_home_rsnapshot/daily.0/ /backups/backup_home_rsnapshot/daily.1/
[29/Nov/2012:13:10:21] mkdir -m 0755 -p /backups/backup_home_rsnapshot/daily.0/
[29/Nov/2012:13:10:21] /usr/bin/rsync -a --delete --numeric-ids -- relative --delete-excluded --link-dest=/backups/backup_home_rsnapshot/daily.1/localhost/ /home/backups/backup_home_rsnapshot/daily.0/localhost/
[29/Nov/2012:13:25:09] touch /backups/backup_home_rsnapshot/daily.0/
[29/Nov/2012:13:25:09] rm -f /var/run/rsnapshot.pid
[29/Nov/2012:13:25:09] /usr/bin/rsnapshot daily: completed successfully
So: 15 minutes... and the changed data amounted to 21GB.
*attic vs. obnam
Not as thorough but makes mention that one of the cons of
obnam is that it's very slow vs.
- well documented
- active mailing list
- packages available
- much smaller backups (even without deduplication)
- much better deduplication
- much faster
- repository format not documented
- not a large user community
Some testing data is shown which would seem to indicate that
obnam is just really slow.
From local SSD to remote HD, over a so-so wifi connection:
rsync: 0:24 0:01
Attic ssh: 0:28 0:05
Attic sshfs: 0:51 0:08
Obnam sftp: 8:45 0:21
Obnam sshfs: 25:22 0:22