Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a list of URLs, and they are all like this: http://www.contratos.gov.co/consultas/detalleProceso.do?numConstancia=10-10-85225 ; inside those pages there are .pdf files being referenced. I need to download those files with wget or curl.

I tried to do this:

wget -r http://www.contratos.gov.co/consultas/detalleProceso.do?numConstancia=10-10-85225 -l2 --accept .pdf

and

wget -r -l1 --no-parent -A ".pdf" http://www.contratos.gov.co/consultas/detalleProceso.do?numConstancia=10-10-85225  

but I get a protocol error:

Resolving www.contratos.gov.co (www.contratos.gov.co)... 201.234.78.2 Connecting to www.contratos.gov.co (www.contratos.gov.co)|201.234.78.2|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] www.contratos.gov.co/consultas/detalleProceso.do?numConstancia=10-10-85225: Protocol error

Cannot write to `www.contratos.gov.co/consultas/detalleProceso.do?numConstancia=10-10-85225' (Protocol error)

What flags can I use to command wget to parse a URL and then download pdf files linked inside of the initial page?

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

Wget and Curl only parse links within the anchor tags on a HTML Document.
The page you are referring to, uses a POST Method with the link to the document to download it.

You will have to download the file and parse it manually for all links. This is something that wget will not do for you.

EDIT: However I do not know why you are receiving a protocol error. Would you mind to run the same commands with a --debug option and paste the output somewhere where we can see it?

share|improve this answer
    
Hi darnir, the protocol error goes away when specifying the output file with -O. You are right, I have to download first the files, then parse them. Thanks for have a look at it! –  Alex. S. Mar 27 '13 at 3:15
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.