Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there an easy way to keep a folder synced with a directory listing via HTTP?

Edit:

Thanks for the tip with wget! I created a shell script and added it as a cron job:

remote_dirs=( "http://example.com/" "…") # Add your remote HTTP directories here
local_dirs=(  "~/examplecom" "…")

for (( i = 0 ; i < ${#local_dirs[@]} ; i++ )) do
cd "${local_dirs[$i]}"
wget -r -l1 --no-parent -A "*.pdf" -nd -nc ${remote_dirs[$i]}
done

# Explanation:
# -r            to download recursively
# -l1           to include only one directory depth
# --no-parent   to exclude parent directories
# -A "*.pdf"    to accept only .pdf files
# -nd           to prevent wget to create directories for everything
# -N            to make wget to download only new files

Edit 2: As mentioned below one could also use --mirror (-m), which is the shorthand for -r -N.

share|improve this question
    
Glad it helped. Could you accept the answer that you feel best helped you resolve your question? –  uther Apr 18 '12 at 22:12
add comment

2 Answers 2

up vote 4 down vote accepted

wget is a great tool.

Use wget -m http://somesite.com/directory

-m
--mirror
    Turn on options suitable for mirroring.  This option turns on
    recursion and time-stamping, sets infinite recursion depth and
    keeps FTP directory listings.  It is currently equivalent to 
    -r -N -l inf --no-remove-listing.
share|improve this answer
add comment

Like rsync, but use zsync to get from an httpd server.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.