Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there an easy way to keep a folder synced with a directory listing via HTTP?


Thanks for the tip with wget! I created a shell script and added it as a cron job:

remote_dirs=( "http://example.com/" "…") # Add your remote HTTP directories here
local_dirs=(  "~/examplecom" "…")

for (( i = 0 ; i < ${#local_dirs[@]} ; i++ )) do
cd "${local_dirs[$i]}"
wget -r -l1 --no-parent -A "*.pdf" -nd -nc ${remote_dirs[$i]}

# Explanation:
# -r            to download recursively
# -l1           to include only one directory depth
# --no-parent   to exclude parent directories
# -A "*.pdf"    to accept only .pdf files
# -nd           to prevent wget to create directories for everything
# -N            to make wget to download only new files

Edit 2: As mentioned below one could also use --mirror (-m), which is the shorthand for -r -N.

share|improve this question
Glad it helped. Could you accept the answer that you feel best helped you resolve your question? – uther Apr 18 '12 at 22:12
up vote 7 down vote accepted

wget is a great tool.

Use wget -m http://somesite.com/directory

    Turn on options suitable for mirroring.  This option turns on
    recursion and time-stamping, sets infinite recursion depth and
    keeps FTP directory listings.  It is currently equivalent to 
    -r -N -l inf --no-remove-listing.
share|improve this answer

Like rsync, but use zsync to get from an httpd server.

share|improve this answer
There is not much documentation for zsync on the Internet. It would be really nice if you could elaborate on your answer. Thank you. – Behrooz Jul 7 '15 at 15:11
Behrooz - I actually use lftp and its mirror command now instead. – gogators Jul 8 '15 at 11:28

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.