Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file.
Is there a way to specify a wget command and then perform a series of functions on the downloaded file before the next file is downloaded?
An example of my script is below:
for FILE in wget -A json -np -nd -r **website URL**
do **somefunctions**
rm $FILE
done
but this doesn't seem to work.
Ended up using the command
wget -O **URL** | awk **file suffix** | cut **delimiter1** | cut **delimiter2** > filenames.txt
then I just iterated through that file to download the files one at a time:
while read FILE; do **commands**; done <filenames.txt
Not extremely elegant, but it did the trick!
-Aoption needs to be something along the lines of-A "*.json"if you wish to download all the.jsonfiles. However sincewgetis non-interactive you will simply download all the files of that type first before it will move on to the next line... You could have a file listing each individual json file url, assigning them to a variable that loops, downloading each individually then completing a function before moving on to the next line in the file. However you would most likely need to manually create such a file... – kemotep Jan 2 '19 at 18:30