0

I signed for Google Drive's Premium 2TB plan for my backups and needed a way to send more than 1 million+ of files there. Web interface is very problematic for this purpose as if the transfer has any errors, identifying what's online and what's not would be very difficult.

So I started to look for some safe way to send files there.

First I found google-drive-ftp-adapter, but as I was trying to install it I got an error that I couldn't solve: "This app is blocked. This app tried to access sensitive info in your Google Account. To keep your account safe Google blocked this access." The project has no activity so I guessed that this was some move by Google that wasn't addressed by the maintainers.

Then I tried to add google drive to Gnome's online accounts. It did mount the drive, but when I tried to upload the files the speed was ridiculous: around 15 KB/s which would take me like 10000 hours to upload the files (around 543GB) and I got some errors even in the first files.

After that I bought Filezilla Pro which connects to Google Drive. So I bought it and it is worked fine. The speed is around 15 MB/s which is not ideal, it will take like 10 hours to upload but that's much better that 10000 and it's doable. Also filezilla has the "Failed transfers" tab that allows me to see the transfers that had an error and I just redo them and that's it.

I wanted a GUI tool, but also gave a chance to rclone that proved to work really well and I could make it reach 32 MB/s which downed the time to 5 hours. The drawback is that if any tranfer has an error I have to cherry pick them in a log file to retransfer them. This is more incovenient than Filezilla.

But none of these solutions felt like my endgame. Each one has it's drawbacks. I really would prefer a FOSS solution preferably with a GUI that simply let me choose the dirs I wanted to copy, redo the errors easily and it's fast enough. GUI tools are preffered but I would consider command line tools if it's really efficient, let me choose what I want and has decent logging while copying preferably visually.

Does anyone has any other solution besides Filezilla Pro and RClone ? How would you send 1 million+ of files from several different folders to Google Drive ?

Edit

I really had underestimated the speed. It's been copying files for 17 hours and there's only 368GB copied. With RClone. 🙄

Edit2

Underestimate ? that's a very far understatement. The upload finally concluded. And guess what: 3d 1h 13m 42.3s to upload 463.169 GiB*. Now... ok I know the small file overhead and stuff. But this overhead level is not acceptable by any standard. I really have to find some way to speed things to an acceptable level.

Searching for a solution I saw that TeraBox has a feature called "cloud decompression". Will give it a try.

*I know the value differs from what I stated above. But that's because I had already uploaded a part of the files in a previous run.

Edit3

TeraBox do have cloud decompression. They do offer 1TB free, their paid plan (2TB) is as cheap as it gets (U$3.49/month) and does have the ability to decompress online. Unfortunately decompression of files larger than 12 Gib isn't supported and to divide 530 Gibs in packages of 12 Gibs I have to cherry pick files in the sub-dirs what is a lot of work. Impractical. So this solution is only applicable if either we want to do that 3 day upload again or maintain subdirs as packages. No FTP client, paid or free, supports TeraBox as Filezilla supports Google Drive. But they do have desktop apps for all major OS (Win, Mac Linux) and a mobile app for Android.

to quote U2: "And I still haven't found, what I'm looking for" :D

2
  • Do you need all your files to be individually accessible on Google Drive, or is an opaque backup OK? (By that I mean a setup where you need to at least partially restore a backup to access individual files.) Commented May 9 at 7:46
  • Well It would be better if they are accessible, but my primary goal is just to prevent data loss, then I think I could consider an opaque backup. You mean like glacier or something ? I checked that options and Google Drive is cheaper. Commented May 9 at 8:51

3 Answers 3

4

I use Déjà Dup Backups for this type of scenario — using a GUI to manage backups to Google Drive. The backups are compressed and encrypted, and Déjà Dup provides clear feedback if things go wrong.

I haven’t checked upload speeds but backup time has never been an issue for me.

2
  • I never used it. But I saw that it's backend, duplicity, uses incremental tar volumes. I will give it a try, but I would rather be able to access the files in google drive directly if I needed. Commented May 9 at 13:44
  • Yes, I didn’t say it explicitly, but since the backups are compressed and encrypted, you can’t access the files directly. Commented May 9 at 15:02
1

check out rsnapshot available from EPEL or https://rsnapshot.org/

I have only used it on local LAN systems to do backups over NFS, but their rsnapshot.conf template file template has this:

###############################
### BACKUP POINTS / SCRIPTS ###
###############################

# LOCALHOST
backup  /home/          localhost/
backup  /etc/           localhost/
backup  /usr/local/     localhost/
#backup /var/log/rsnapshot              localhost/
#backup /etc/passwd     localhost/
#backup /home/foo/My Documents/         localhost/
#backup /foo/bar/       localhost/      one_fs=1,rsync_short_args=-urltvpog
#backup_script  /usr/local/bin/backup_pgsql.sh  localhost/postgres/
# You must set linux_lvm_* parameters below before using lvm snapshots
#backup lvm://vg0/xen-home/     lvm-vg0/xen-home/

# EXAMPLE.COM
#backup_exec    /bin/date "+ backup of example.com started at %c"
#backup [email protected]:/home/ example.com/    +rsync_long_args=--bwlimit=16,exclude=core
#backup [email protected]:/etc/  example.com/    exclude=mtab,exclude=core
#backup_exec    ssh [email protected] "mysqldump -A > /var/db/dump/mysql.sql"
#backup [email protected]:/var/db/dump/  example.com/
#backup_exec    /bin/date "+ backup of example.com ended at %c"

# CVS.SOURCEFORGE.NET
#backup_script  /usr/local/bin/backup_rsnapshot_cvsroot.sh      rsnapshot.cvs.sourceforge.net/

# RSYNC.SAMBA.ORG
#backup rsync://rsync.samba.org/rsyncftp/       rsync.samba.org/rsyncftp/

on a 1 gbps wired LAN I consistently get 112 MB/s. But going over the internet you will be at the mercy of many other things outside your control. My experience though has been that ssh or scp is one of the fastest modes of transfer, when they are not many small files where the I/O of your operating system and file system will then be the problem. I find that when some folder has many files and subfolders that are not of sufficient size to maximize transfer speed that doing tar -cf on the system first then transferring 1 large tar file is the best way, then untarring it at the destination. And for backups specifically sometimes it's better to just have all tar files on your back up system, it's subjective depending on what your goal is.

1
  • I haven't tried it... but AFAIU it works with ssh connections. How would I access Google Drive to send files ? Commented May 9 at 17:47
1

I'm using the OneDrive client available from https://github.com/abraunegg/onedrive/. Although is available in most Linux distribution repositories the author has a large message warning people only to use the latest version, which right now is "v2.4.25-1+np2+4.1".

I know that OneDrive is Microsoft's offering, whereas you are using Google Drive, but I don't have a Google Drive offering to test. Perhaps you can use the process outlined here to perform your own tests and share your conclusions.

Empirically, and without significant testing, what I'm seeing is that there is a significant per-file overhead, but the transfer speed is pretty fast. For example,

ls -lh ???_TEMPORARY.*
-rw-rw-r-- 1 chris  29 May  9 17:49 DTE_TEMPORARY.txt
-rwxrwxr-- 1 chris 68M Jun 11  2016 VID_TEMPORARY.mp4

The text file took around one second to upload. The video took around six seconds:

May 09 17:48:16 onedrive[316481]: Uploading new file ./VID_TEMPORARY.mp4 ...
May 09 17:48:22 onedrive[316481]: done.
May 09 17:49:12 onedrive[316481]: Uploading new file ./DTE_TEMPORARY.txt ...
May 09 17:49:13 onedrive[316481]: done.

I felt more analysis was required, so I copied the text file 100 times and measured the overall copy time:

May 09 17:56:29 onedrive[316481]: Uploading new file ./TEMPORARY/DTE_1.txt ...
May 09 17:56:30 onedrive[316481]: done.
…
May 09 17:57:59 onedrive[316481]: Uploading new file ./TEMPORARY/DTE_100.txt ...
May 09 17:57:59 onedrive[316481]: done.

And then the same for 1 copy of the video concatenated to itself 10 times (i.e. 677MB):

May 09 17:59:28 onedrive[316481]: Uploading new file ./TEMPORARY/VID_HUGE.mp4 ...
May 09 18:00:24 onedrive[316481]: done.

Results:

  • 1 minute 30 seconds for 100 tiny files (2900 bytes total). 32 bytes/sec
  • 26 seconds for 1 huge (677MB) file. Average speed: Average speed: 26 MB/s

Having also used S3 FUSE filesystems I would suggest that the per-file overhead is what is hitting your overall file transfer rate with Google Drive

8
  • Onedrive more than doubles the price. The 1TB plan is in Reals (Brazil's currency, where I live) R$ 36/month and an extra 1Tb for R$ 45 (R$ 81 total). The 2TB plan in Google Drive is R$ 38,99. Also I already use Google Drive a lot. I used to sign the 200GB plan. But thanks for the suggestion. It may be useful for somebody else. Commented May 9 at 17:36
  • I wasn't suggesting switching to OneDrive. It was more about the relative speeds for many small files vs fewer much larger files Commented May 9 at 18:24
  • 1
    (1/2) Something that StephenKitt asked was whether you needed all the files individually available, @NelsonTeixeira. If you could tar (zip) directories of files you'd get fewer but larger files to upload, potentially increasing the overall throughput. You could write some code that created a tar/zip file for each directory (without its children), resulting in one upload per directory rather than one upload per file Commented May 10 at 8:10
  • 1
    (2/2) I used to do something similar when backing up to "hubiC" - it had a five second penalty per upload, but once running the upload speed was really quite respectable Commented May 10 at 8:11
  • 1
    Depending on what you're after, run a tarball per directory (or tree of directories) so that you have a backup. Then over time and within reason without caring about how long it takes replace each tarball with the individual files. It really depends on how complex you want to get Commented May 10 at 17:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .