0

I'm trying to use the FCC's Electronic Comment Filing System (ECFS) to bulk download filings in individual proceedings. They have an API that will return every filing in a proceeding. It returns a URL for individual documents in the format:

https://www.fcc.gov/ecfs/document/10809709027819/1

However, while this works in the browser, it only downloads a placeholder HTML file saying JavaScript it required when I use wget or curl. I tried examining the page in my browser but couldn't find anything like a source URL for the actual PDF.

Is there a way to use wget or curl to get at the actual PDF?

Improve this question

1 Answer 1

Reset to default
1

I took a look at the request that returns the PDF using Firefox's browser tools, and trimmed down the request headers to get the minimal request that returns a PDF. This works for me:

% curl 'https://www.fcc.gov/ecfs/documents/10809709027819/1' -H 'Referer: https://www.fcc.gov/ecfs/document/10809709027819/1' | file -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  146k    0  146k    0     0   516k      0 --:--:-- --:--:-- --:--:--  518k
/dev/stdin: PDF document, version 1.7, 9 page(s)

So at least for this PDF, it looks like all that's needed is to set the Referer header to the same URL.

Improve this answer
2
  • 1
    You don't even need the Referer; Notice there's a slight difference between the URL (documents) and the Referer (document). So it works with the plural form (documents instead of document) without the Referer.
    – aviro
    Aug 14 at 10:21
  • Huh, I didnt notice that all, which is why I didn't try eliminating that header. Thanks!
    – muru
    Aug 14 at 10:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .