How to make multiple parallel calls to function which makes post request in unix?

Question

I am having a function A, which takes an argument fileName and then makes a curl post call with that file to employment Server.The function pseudocode will look as :

function A(filename)
{
// read file from local 
// send it to deployment server and via curl/any rest client
// read response code
// store data of filename and response code 
}

for all files f in folder :
 A(f)

// See all response code and print data, which all files passed and which all failed

Now if i want to run in parallel I will use :

A(f) & inside for loop

I have concerns in highlighted part, how can I gather details of filename,response code,since there is no hashmap/dictionary type of support in unix easily available. Also if I print the data in A, then if I keep two lines one to print filename and one to print response code then two lines may not be sequential because of parallel nature. Although I can print both in one line,but I will like to capture results to do some processing later.Is there a easy way around?

Answer 1

I would use parset:

a() {
  filename="$1"
  echo "***  Here    is  ***"
  sleep 0.$RANDOM
  echo "the response code to"
  sleep 0.$RANDOM
  echo "$filename"
}
export -f a

# Put the results into a bash array. The order will be kept.
# Here we use "File  a" and "File  b" as input. Change those to your needs
parset res a ::: "File  a" "File  b"
echo "${res[0]}"
echo "${res[1]}"

Alternatively:

# If you want the output into an assoc array, convert the results:
# First build an array of the input
input=("File  a" "File  b")
# Then run the jobs in parallel
parset res a ::: "${input[@]}"

# Finally zip the two arrays to a single associative array
declare -A myassoc
for ((i=0; $i<${#input[@]}; i++)); do
  myassoc[${input[i]}]=${res[i]}
done
echo "${myassoc["File  a"]}"

parset is part of GNU Parallel.

To activate parset you need to activate it as a function. The function is defined as part of env_parallel.

Do the below and restart your shell.

bash:  Put this in $HOME/.bashrc:  . `which env_parallel.bash`
       E.g. by doing:  echo '. `which env_parallel.bash`' >> $HOME/.bashrc
       Supports: aliases, functions, variables, arrays

zsh:   Put this in $HOME/.zshrc:  . `which env_parallel.zsh`
       E.g. by doing:  echo '. `which env_parallel.zsh`' >> $HOME/.zshenv
       Supports: functions, variables, arrays

fish:  Unsupported

ksh:   Put this in $HOME/.kshrc:  source `which env_parallel.ksh`
       E.g. by doing:  echo 'source `which env_parallel.ksh`' >> $HOME/.kshrc
       Supports: aliases, functions, variables, arrays

mksh:  Put this in $HOME/.mkshrc:  source `which env_parallel.mksh`
       E.g. by doing:  echo 'source `which env_parallel.mksh`' >> $HOME/.mkshrc
       Supports: aliases, functions, variables, arrays

pdksh: Put this in $HOME/.profile:  source `which env_parallel.pdksh`
       E.g. by doing:  echo '. `which env_parallel.pdksh`' >> $HOME/.profile
       Supports: aliases, functions, variables, arrays

ash:   Put this in $HOME/.profile:  . `which env_parallel.ash`
       E.g. by doing:  echo '. `which env_parallel.ash`' >> $HOME/.profile
       Supports: aliases, variables

dash:  Put this in $HOME/.profile:  . `which env_parallel.dash`
       E.g. by doing:  echo '. `which env_parallel.dash`' >> $HOME/.profile
       Supports: aliases, variables

csh:   Unsupported

tcsh:  Unsupported

To install in all shells run:

  parset --install