2

I am having a function A, which takes an argument fileName and then makes a curl post call with that file to employment Server.The function pseudocode will look as :

function A(filename)
{
// read file from local 
// send it to deployment server and via curl/any rest client
// read response code
// store data of filename and response code 
}

for all files f in folder :
 A(f)

// See all response code and print data, which all files passed and which all failed

Now if i want to run in parallel I will use :

A(f) & inside for loop

I have concerns in highlighted part, how can I gather details of filename,response code,since there is no hashmap/dictionary type of support in unix easily available. Also if I print the data in A, then if I keep two lines one to print filename and one to print response code then two lines may not be sequential because of parallel nature. Although I can print both in one line,but I will like to capture results to do some processing later.Is there a easy way around?

4
  • Bash has arrays (and even associative arrays), so there is your hashmap/dictionary support, easily available. I am not following the part about your worries with sequentiality; just capture the results if you want to process them later.
    – dirkt
    Jun 15 '21 at 20:12
  • I will read about associative arrays. i did not get your second statement. I want to run all fxn calls in parallel not sequentially. Jun 15 '21 at 20:21
  • So run them in parallel, and collect the results (e.g. in an array). Once they've finished, process the results. Where is the problem?
    – dirkt
    Jun 15 '21 at 20:37
  • I was thinking that how will i store filename to response code mapping,as i was not aware of associative arrays in shell script. I thought i might have to find some other solution.Now as there are associative arrays that i get to know as told, it looks okay now. Jun 15 '21 at 21:00
1

I would use parset:

a() {
  filename="$1"
  echo "***  Here    is  ***"
  sleep 0.$RANDOM
  echo "the response code to"
  sleep 0.$RANDOM
  echo "$filename"
}
export -f a

# Put the results into a bash array. The order will be kept.
# Here we use "File  a" and "File  b" as input. Change those to your needs
parset res a ::: "File  a" "File  b"
echo "${res[0]}"
echo "${res[1]}"

Alternatively:

# If you want the output into an assoc array, convert the results:
# First build an array of the input
input=("File  a" "File  b")
# Then run the jobs in parallel
parset res a ::: "${input[@]}"

# Finally zip the two arrays to a single associative array
declare -A myassoc
for ((i=0; $i<${#input[@]}; i++)); do
  myassoc[${input[i]}]=${res[i]}
done
echo "${myassoc["File  a"]}"

parset is part of GNU Parallel.

To activate parset you need to activate it as a function. The function is defined as part of env_parallel.

Do the below and restart your shell.

bash:  Put this in $HOME/.bashrc:  . `which env_parallel.bash`
       E.g. by doing:  echo '. `which env_parallel.bash`' >> $HOME/.bashrc
       Supports: aliases, functions, variables, arrays

zsh:   Put this in $HOME/.zshrc:  . `which env_parallel.zsh`
       E.g. by doing:  echo '. `which env_parallel.zsh`' >> $HOME/.zshenv
       Supports: functions, variables, arrays

fish:  Unsupported

ksh:   Put this in $HOME/.kshrc:  source `which env_parallel.ksh`
       E.g. by doing:  echo 'source `which env_parallel.ksh`' >> $HOME/.kshrc
       Supports: aliases, functions, variables, arrays

mksh:  Put this in $HOME/.mkshrc:  source `which env_parallel.mksh`
       E.g. by doing:  echo 'source `which env_parallel.mksh`' >> $HOME/.mkshrc
       Supports: aliases, functions, variables, arrays

pdksh: Put this in $HOME/.profile:  source `which env_parallel.pdksh`
       E.g. by doing:  echo '. `which env_parallel.pdksh`' >> $HOME/.profile
       Supports: aliases, functions, variables, arrays

ash:   Put this in $HOME/.profile:  . `which env_parallel.ash`
       E.g. by doing:  echo '. `which env_parallel.ash`' >> $HOME/.profile
       Supports: aliases, variables

dash:  Put this in $HOME/.profile:  . `which env_parallel.dash`
       E.g. by doing:  echo '. `which env_parallel.dash`' >> $HOME/.profile
       Supports: aliases, variables

csh:   Unsupported

tcsh:  Unsupported

To install in all shells run:

  parset --install
4
  • what does "File a" means? Jun 21 '21 at 21:52
  • I installed "brew install parallel" on my system and now running parset is giving error.Unknown Option : res Unknown option: a ....................................................... parset only works if it is a function. The fucntion is defined as a part of env_parallel Jun 21 '21 at 22:29
  • 1
    @rahulsharma The answer is in the part you left out.
    – Ole Tange
    Jun 22 '21 at 8:10
  • do I need to add some fxn name in bash_profile?I did not actually get what that part is saying.Would u be able to tell me plz? Jun 23 '21 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.