I'm building self-driving robot from scratch and in order to minimize cost and maximize battery life, I decided to stream video to my PC from camera installed on robot, using Raspberry Pi Zero W, and then perform all ML computations on PC. As for now, using Gstreamer I'm getting delays around 100ms, which should be fine, especially on low speed driving. On PC side, video stream is received by OpenCV VideoCapture and then passed as input to ML part.
My current Gstreamer commands looks as following:
- Raspberry Pi:
raspivid -t 0 -hf -n -h 240 -w 320 -fps 60 -o - | gst-launch-1.0 -e -v fdsrc ! h264parse ! rtph264pay pt=96 config-interval=5 ! udpsink host=<IP of my PC> port=5000
- PC (this command works from console, OpenCV uses different final sink):
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp, payload=96 ! rtpjitterbuffer ! rtph264depay ! avdec_h264 ! xvimagesink sync=false
However to make ML running I need some additional data (few bytes) from various sensors. I'm thinking about sending that data along with video, so that each video frame would have assigned corresponding sensor data. My question is what is the best way to accomplish that. From what I found, it looks like it should be possible by writing my own Gstreamer plugin, but maybe there is some easier way to do it? Second thing is about receiving that data on the other end. Is it possible with OpenCV VideoCapture, or do I need accomplish it in some other way?