2

I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file.

Is there a way to specify a wget command and then perform a series of functions on the downloaded file before the next file is downloaded?

An example of my script is below:

for FILE in wget -A json -np -nd -r **website URL**
  do **somefunctions** 
  rm $FILE
done

but this doesn't seem to work.

3
  • 1
    Your -A option needs to be something along the lines of -A "*.json" if you wish to download all the .json files. However since wget is non-interactive you will simply download all the files of that type first before it will move on to the next line... You could have a file listing each individual json file url, assigning them to a variable that loops, downloading each individually then completing a function before moving on to the next line in the file. However you would most likely need to manually create such a file...
    – kemotep
    Jan 2, 2019 at 18:30
  • that's what I figured... is there a way to list all of the file names using wget without actually downloading them? Can't seem to find anything in the help menu or by googling...
    – lstbl
    Jan 2, 2019 at 20:15
  • 1
    You would have to use curl and pipe the results to a file. But the website you are "scrapping" for these results could block directory listing for security reasons so you will have to copy and paste urls manually if they prevent this.
    – kemotep
    Jan 2, 2019 at 20:21

1 Answer 1

1

Ended up using the command

wget -O **URL** | awk **file suffix** | cut **delimiter1** | cut **delimiter2** > filenames.txt

then I just iterated through that file to download the files one at a time:

while read FILE; do **commands**; done <filenames.txt

Not extremely elegant, but it did the trick!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .