Creating a process for each redirected stdout

Question

So, I've got a port scanner (masscan), and I want to redirect its output to a certain command-line which is going to check if every resulting IP:Port combination serves a certain web service.

So, it can go like this:

masscan 0.0.0.0/0 -p8080 --max-rate 300000 | ./scanner.py

But this would be slow for huge IP ranges, since scanner.py is going to process one-by-one.

How can I do this in parallel? i.e redirecting each IP on its own and creating processes that will work in parallel processing the stdin that was redirected to them, thus making the scan faster.

Answer 1

I'd think you can do something with xargs. For example the following:

masscan 0.0.0.0/0 -p8080 --max-rate 300000 | \
    xargs -L1 -I+ -P200 sh -c "echo '+' | ./scanner.py"

That would run up to 200 parallel ./scanner.py, each one handling a single result line from masscan.