2

I am gathering some data locally at home on a Raspberry Pi and I want to send the data as fast as possible to a REST API (which I own) on a server on the web.

The locally gathered data could flow as fast as 100 records per second.

If I execute a curl command in a loop it will send the data... Wait for the 200 response then process the next record... Much slower than my internal data flow.

I have found some hints here on Stackoverflow and tried to adapt them but it wouldn't send the curl commands in parallel.

I know my code is not the prettiest (particularly about the usage of the mycurl function ) and I am ready for suggestions

#!/bin/bash

host="localhost"
port="********"

mycurl() {
    data="field1=${1}&field2=${2}&field3=${3}&field4=${4}&field5=${5}&field6=${6}&field7=${7}&field8=${8}&field9=${9}&field10=${10}"
   curl --output /dev/null -d $data --silent -X POST https://myapi/myendpoint;
}
export -f mycurl

#----------------------LOOP starts------------------------
while true; 
do 

   nc -d $host $port | while IFS="," read -r f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 f14 f15 f16 f17 f18 f19 f20 f21 f22
   do 
      if [ "$f15" != "" ]; then 
          seq 1000  | parallel --no-notice --joblog log -j0 mycurl ::: ${f5} ::: ${f7} ::: ${f8} ::: ${f15} ::: ${f16} ::: ${17} ::: ${18} ::: ${19} ::: ${20} ::: ${21};
      fi
   done

done
0

2 Answers 2

1

GNU Parallel has an overhead of 2-5 ms per job which is getting close to your 100 entries per second. To mitigate that you can wrap it with parallel --pipe. This should run around 25000 jobs in parallel per second on a 100 core machine:

#!/bin/bash

host="localhost"
port="********"

mycurl() {
  if [ "${15}" != "" ]; then
    # The numbering here should probably be adjusted
    data="field1=${1}&field2=${2}&field3=${3}&field4=${4}&field5=${5}&field6=${6}&field7=${7}&field8=${8}&field9=${9}&field10=${10}"
    curl --output /dev/null -d $data --silent -X POST https://myapi/myendpoint;
  fi
}
export -f mycurl

#----------------------LOOP starts------------------------
while true; 
do 
  nc -d $host $port
done |
  parallel -j100 --pipe --block 100k parallel --colsep , -j0 mycurl
0

Why not fork each curl call to the background: curl... &. Putting an ampersand at the end of a shell command is like using a ; semi-colon, only it forks the process to the background.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .