I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file.

Is there a way to specify a wget command and then perform a series of functions on the downloaded file before the next file is downloaded?

An example of my script is below:

for FILE in wget -A json -np -nd -r **website URL**
  do **somefunctions** 
  rm $FILE

but this doesn't seem to work.

  • 1
    Your -A option needs to be something along the lines of -A "*.json" if you wish to download all the .json files. However since wget is non-interactive you will simply download all the files of that type first before it will move on to the next line... You could have a file listing each individual json file url, assigning them to a variable that loops, downloading each individually then completing a function before moving on to the next line in the file. However you would most likely need to manually create such a file... – kemotep Jan 2 '19 at 18:30
  • that's what I figured... is there a way to list all of the file names using wget without actually downloading them? Can't seem to find anything in the help menu or by googling... – lstbl Jan 2 '19 at 20:15
  • 1
    You would have to use curl and pipe the results to a file. But the website you are "scrapping" for these results could block directory listing for security reasons so you will have to copy and paste urls manually if they prevent this. – kemotep Jan 2 '19 at 20:21

Ended up using the command

wget -O **URL** | awk **file suffix** | cut **delimiter1** | cut **delimiter2** > filenames.txt

then I just iterated through that file to download the files one at a time:

while read FILE; do **commands**; done <filenames.txt

Not extremely elegant, but it did the trick!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.