5

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following:

$ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \
    --password='password' https://cloud.domain.com/foo/carddav

The total number of .vcf files is about 400, after downloading about 70 of them, wget returns this error:

   ss of error

original URL: http://oi40.tinypic.com/2ch9itt.jpg

Which kills the process because the system is "out of memory". The system is a Debian Wheezy virtual machine, hosted on Windows 7. I tried to raise the RAM to 1024MB instead of the actual 128MB, but the problem still exists.

Any suggestions on how to work around this or alternative ways to accomplish this?

  • HTTrack might be an alternative to wget in your case. It is available in Debian. – jofel Jan 7 '14 at 14:13
  • 1
    wget was consuming about 530MB of memory at the time it was killed, so that's very abnormal. Could you try with curl? – mveroone Jan 7 '14 at 14:38
  • What is this -f switch? wget doesn't seem to have such an option. – psusi Jan 7 '14 at 15:24
  • @jofel: thank you for the suggestion, I will use it as a last option if I can't do it with wget. – giovi321 Jan 7 '14 at 16:22
  • @psusi: there is no -f switch in my command, probably you saw the -A switch followed by the .vcf extension, without the dot. It is "-Avcf", where vcf is the argument of the switch -A. – giovi321 Jan 7 '14 at 16:23
3

It seems wget loops causes memory overflow.
The natural first suggestion is to increase again memory of your cloud instance from 1Gb to 2Gb. This solved a similar issue recently.

If this is not possible or doesn't solve the problem the second solution is to run wget within 2 steps:

  1. Retrieve files list. As I see in your screenshot the files are in the directory cloud.some_domain.com/remote.php/carddav/addressbooks/your_name/.
    So, run wget to get the directory index:
    wget https://cloud.some_domain.com/remote.php/carddav/addressbooks/your_name/
    This will give you an index.html file.

  2. Now you can parse it to retrieve the filenames to download:

    grep ".vcf" index.html | awk -F"href=" '{print $2}' | awk -F\" '{print $2}' > ALL_VCF_FILES.lst
    for elt in `cat ALL_VCF_FILES.lst`
    do
    wget https://cloud.some_domain.com/remote.php/carddav/addressbooks/your_name/$elt
    done
    rm ALL_VCF_FILES.lst index.html
    
  • thank you very much that's the best solution ti my problem! I'll let you know asap if it works. – giovi321 Jan 7 '14 at 16:24
  • You welcome ! I edited the code parts to match best your case (HTTPS) – Slyx Jan 7 '14 at 16:30
  • I have to thank @Slyx! Your code was almost perfect. The only problem was that the ALL_VCF_FILES.lst list was a list like /remote.php/carddav/addressbooks/user/contacts/xxxxxxxxx.vcf this means that the link is missing the "cloud.domain.com". Using sed I made a simple find and replace like this: sed 's/\/remote.php\/carddav\/addressbooks\/user\/contacts\//https:\/\/www.cloud.domain.com\/remote.php\/carddav\/addressbooks\/user\/contacts\//g' ALL_VCF_FILES.lst > all_vcf_files.lst Maybe you can incorporate this in your answer to make the code complete. It works like a charm! – giovi321 Jan 7 '14 at 16:48
  • wget --no-check-certificate -e robots=off --user=user --password='pswd' blabla grep ".vcf" index.html | awk -F"href=" '{print $2}' | awk -F\" '{print $2}' > ALL_VCF_FILES.lst sed 's/\/remote.php\/carddav\/addressbooks\/user\/contacts\//https:\/\/cloud.domain.com\/remote.php\/carddav\/addressbooks\/user\/contacts\//g' ALL_VCF_FILES.lst > all_vcf_files.lst for elt in cat all_vcf_files.lst do wget -nc --no-check-certificate -e robots=off --user=user --password='pswd' $elt done rm ALL_VCF_FILES.lst index.html all_vcf_files.lst mv *.vcf /home/scripts/owncloud_backup/vcf/ – giovi321 Jan 7 '14 at 16:51
  • @user3161330 instead of adding a sed line we can just add the missing part of the URL before the $elt variable. See my edit. 2nd : you have the pathchain in your index.html because you requested this specific file in your wget commad. – Slyx Jan 7 '14 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.