New answers tagged parallel
There are at least two ways of speeding up grep CPU-wise: If you're searching for a fixed string rather than a regular expression, specify the -F flag; If your pattern is ASCII-only, use an 8-bit locale instead of UTF-8, e.g. LC_ALL=C grep .... These won't help though if your hard drive is the bottleneck; in that case probably parallelizing won't help ...
If the problem is not I/O bound you could use a tool that is optimized for multi-core processing. You might want to take a look at sift (http://sift-tool.org, disclaimer: I am the author of this tool) or the silver searcher (https://github.com/ggreer/the_silver_searcher). the silver searcher has a file size limit of 2GB if you use a regex pattern and not a ...
Why not use --pipe AND --pipepart with GNU Parallel? This eliminates the extra cat and starts direct reads from the file on disk: parallel --pipe --pipepart -a 2011.psv --block 500M ./carga_postgres.sh
There are two easy solutions for this. Basically, using xargs or parallel. xargs Approach: You can use xargs with find as follows: find . -type f -print0 | xargs -0 -P number_of_processes grep mypattern > output Where you will replace number_of_processes by the maximum number of processes you want to be launched. However, this is not guaranteed to ...
Top 50 recent answers are included