0

I want to use parallel in this example:

cat codigos.txt | parallel -j 10 "wget http://mywebsite.com/teste?id={}&x=&date=01/01/2017" --load-cookies=cookies.txt

I have three files: codigos.txt, a file with the ids, dates.txt, a file with the dates and cookies.txt the file with the cookie that I need to do the request.

What I want to do is to save the response from mywebsite.com for each url, using the contents of my files.

Problems:

  1. How do I do to use information coming from two files?
  2. The url is cut at the '&' (even when I put the quotes). Example:

request made to http://mywebsite.com/teste?id=1

As this example (but without put the space or cut because of the '&'):

cat abc-file | parallel -a - -a def-file echo

reference: https://www.gnu.org/software/parallel/parallel_tutorial.html

5
  • I forgot to say, but the request just goes until the &, how do I say that the & is a character and not a command? Jun 12, 2017 at 22:11
  • The main thrust of your question is very unclear, but as to the other, enclose the URL in 'strong quotes', which prevents the &s from being parsed by the shell.
    – DopeGhoti
    Jun 12, 2017 at 22:16
  • I have two files: - codigos.txt: its a file with the ids; - dates.txt: its a file with the dates; What I want to do is: -Save the response from mywebsite.com for each url. Jun 12, 2017 at 22:21
  • And the 'strong quotes' are not working, I tried with another website without using the parallel (ex: wget mywebsite.com/…. But it doesn't work when I get the code from a file. Jun 12, 2017 at 22:30
  • mywget() { wget "http://mywebsite.com/teste?id=$1&x=&date=$2" --load-cookies=cookies.txt; }; export -f mywget; parallel mywget :::: codigos.txt dates.txt
    – Ole Tange
    Jun 13, 2017 at 19:06

1 Answer 1

0

I assume you want to run:

wget http://mywebsite.com/teste?id=codigo1&x=&date=01/01/2017" --load-cookies=cookies.txt
wget http://mywebsite.com/teste?id=codigo1&x=&date=02/01/2017" --load-cookies=cookies.txt
wget http://mywebsite.com/teste?id=codigo1&x=&date=03/01/2017" --load-cookies=cookies.txt
wget http://mywebsite.com/teste?id=codigo2&x=&date=01/01/2017" --load-cookies=cookies.txt
wget http://mywebsite.com/teste?id=codigo2&x=&date=02/01/2017" --load-cookies=cookies.txt
wget http://mywebsite.com/teste?id=codigo2&x=&date=03/01/2017" --load-cookies=cookies.txt

if the file codigos.txt contains:

codigo1
codigo2

and dates.txt contains:

01/01/2017
02/01/2017
03/01/2017

then this will work:

mywget() {
    wget "http://mywebsite.com/teste?id=$1&x=&date=$2" --load-cookies=cookies.txt;
}
export -f mywget
parallel mywget :::: codigos.txt dates.txt

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .