I have a large dataset (>200k files) that I would like to process (convert files into another format). The algorithm is mostly single-threaded, so it would be natural to use parallel processing. However, I want to do an unusual thing. Each file can be converted using one of two methods (CPU- and GPU-based), and I would like to utilize both CPU and GPU at the same time.
Speaking abstractly, I have two different commands (foo
and bar
), which are supposed to produce equivalent results. I would like to organize two thread pools with fixed capacity that would run up to N instances of foo
and M instances of bar
respectively, and process each input file with either of those pools depending on which one has free slots (determinism is not required or desired).
Is it possible to do something like that in GNU parallel or with any other tool?