1

I have a bash function that mainly curls an endpoint for a set of links, and again curls each of those links (for another set of links) recursively.

task() {
  link="$1"
  response=$(curl "$link")
  
  # save response data to file...

  # process response to find another set of links and save to variable x...
  if condition
      echo "$x" | while read -r link
        task "$link"
}

The above code yields to ~100 curl operations which take a long time. Mainly the endpoint's response time (not processing) is the bottleneck.

Conditions

  1. We have 0 info about how many links are there. We can't have a child link without curling it parent link first.
  2. Each response of curl link should be atomically saved to a file

Is it even possible to parallelize this procedure? I'm OK if the solution needs (or is more elegant with) GNU parallel or some other external tool.

4
  • Does each curl produce multiple child links (in which case they might be parallelizable), or just one (in which case they can only be sequential)? Also, if the network is the bottleneck, would parallelization actually help at all (since multiple simultaneous curls will just compete for bandwidth)? Sep 3, 2021 at 7:39
  • @GordonDavisson Each curl produces multiple links. By network, I meant the API's GET response time. I edit my post now. Sep 3, 2021 at 7:47
  • 1
    External tools like GNU parallel or xargs -P won't work with a function (not directly, anyway - you could make it run sh -c 'task ...' with a .bashrc that defined the function) because the bash function only exists within the bash shell where it is defined. They will work with scripts that implement the same function, same as they would with any other executable. If you really want to do this with a function, you could just run the task function in the background with & but you'd have to write your own queueing code to limit the number of simultaneous tasks.
    – cas
    Sep 3, 2021 at 8:48
  • As @GordonDavisson says, though, since CPU isn't the bottleneck, parallelisation is unlikely to help and may even harm performance.
    – cas
    Sep 3, 2021 at 8:49

1 Answer 1

1

Can you use the breadth-first spider as a template: https://www.gnu.org/software/parallel/man.html#example-breadth-first-parallel-web-crawler-mirrorer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .