67

I'm not entirely sure what you're asking here. Yes, top shows CPU usage as a percentage of a single CPU by default. That's why you can have percentages that are >100. On a system with 4 cores, you can see up to 400% CPU usage. You can change this behavior by pressing I (that's Shift + i and toggles "Irix mode") while top is running. That will cause it to ...


60

The common protocols HTTP, FTP and SFTP support range requests, so you can request part of a file. Note that this also requires server support, so it might or might not work in practice. You can use curl and the -r or --range option to specify the range and eventually just catting the files together. Example: curl -r 0-104857600 -o distro1.iso '...


46

As mavillan already suggested, just use terminator. It allows to display many terminals in a tiled way. When enabling the broadcasting feature, you can enter the very same command simultaneously on each terminal. Here is an example with the date command broadcasted to a grid of 32 terminals.


34

nproc gives the number of CPU cores/threads available, e.g. 8 on a quad-core CPU supporting two-way SMT. The number of jobs you can run in parallel with make using the -j option depends on a number of factors: the amount of available memory the amount of memory used by each make job the extent to which make jobs are I/O- or CPU-bound make -j$(nproc) is a ...


33

If you have a copy of xargs that supports parallel execution with -P, you can simply do printf '%s\0' *.png | xargs -0 -I {} -P 4 ./pngout -s0 {} R{} For other ideas, the Wooledge Bash wiki has a section in the Process Management article describing exactly what you want.


32

There are two easy solutions for this. Basically, using xargs or parallel. xargs Approach: You can use xargs with find as follows: find . -type f -print0 | xargs -0 -P number_of_processes grep mypattern > output Where you will replace number_of_processes by the maximum number of processes you want to be launched. However, this is not guaranteed to ...


28

Check out pee ("tee standard input to pipes") from moreutils. This is basically equivalent to Marco's tee command, but a little simpler to type. $ echo foo | pee md5sum sha256sum d3b07384d113edec49eaa6238ad5ff00 - b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c - $ pee md5sum sha256sum <foo.iso f109ffd6612e36e0fc1597eda65e9cf0 - ...


27

GNU Parallel is designed for this kind of tasks: parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output ::: *.input or: ls | parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output It will run one jobs per CPU core. You can install GNU Parallel simply by: wget https://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel ...


27

Use wait. For example: Data1 ... > Data1Res.csv & Data2 ... > Data2Res.csv & wait AnalysisProg will: run the Data1 and Data2 pipes as background jobs wait for them both to finish run AnalysisProg. See, e.g., this question.


26

On linux, the system call to set the CPU affinity for a process is sched_setaffinity. Then there's the taskset tool to do it on the command line. To have that single program run on only one CPU, I think you'd want something like taskset -c 1 ./myprogram (set any CPU number as an argument to the -c switch.) That should be close enough to a single-...


24

This looks like a job for gnu parallel: parallel bash -c ::: script_* The advantage is that you don't have to group your scripts by cores, parallel will do that for you. Of course, if you don't want to babysit the SSH session while the scripts are running, you should use nohup or screen


23

Using GNU Parallel, $ parallel -j ${jobs} wget < urls.txt or xargs from GNU Findutils, $ xargs -n 1 -P ${jobs} wget < urls.txt where ${jobs} is the maximum number of wget you want to allow to run concurrently (setting -n to 1 to get one wget invocation per line in urls.txt). Without -j/-P, parallel will run as many jobs at a time as CPU cores (...


20

Confiq's answer is a good one for small i and j. However, given the size of i and j in your question, you will likely want to limit the overall number of processes spawned. You can do this with the parallel command or some versions of xargs. For example, using an xargs that supports the -P flag you could parallelize your inner loop as follows: for i in {...


17

If you're using GNU xargs, there's --process-slot-var: --process-slot-var=environment-variable-name Set the environment variable environment-variable-name to a unique value in each running child process. Each value is a decimal integer. Values are reused once child processes exit. This can be used in a rudimentary load distribution scheme, for ...


16

Yes, it is. If you want to do two things concurrently, and wait for them both to complete, you can do something like: sh ./stay/get_it_ios.sh & PIDIOS=$! sh ./stay/get_it_mix.sh & PIDMIX=$! wait $PIDIOS wait $PIDMIX Your script will then run both scripts in parallel, and wait for both scripts to complete before continuing.


16

for((i=1;i<100;i++)); do nohup bash script${i}.sh & done


16

$! is guaranteed to give you the pid of the process in which the shell executed that tail command. Shells are single threaded, each shell lives in its own process with its own set of variables. There's no way the $! of one shell is going to leak into another shell, just like assigning a shell variable in one shell is not going to affect the variable of the ...


14

A problem with split --filter is that the output can be mixed up, so you get half a line from process 1 followed by half a line from process 2. GNU Parallel guarantees there will be no mixup. So assume you want to do: A | B | C But that B is terribly slow, and thus you want to parallelize that. Then you can do: A | parallel --pipe B | C GNU Parallel ...


14

A better way would be to use GNU Parallel. GNU parallel is simple and with it we can control the number of jobs to run in parallel with more control over the jobs. In the below command, script{1..3}.sh gets expanded and are sent as arguments to bash in parallel. Here -j0 indicates that as many jobs should be run as possible. By default parallel runs one job ...


13

When you write A | B, both processes already run in parallel. If you see them as using only one core, that's probably because either of CPU affinity settings (perhaps there is some tool to spawn a process with different affinity) or because one process isn't enough to hold a whole core, and the system "prefers" not to spread out computing. To run several B'...


12

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel: ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > {}res....


12

Red represents the time spent in the kernel, typically processing system calls on behalf of processes. This includes time spent on I/O. There’s no point in trying to reduce it just for the sake of reducing it, because it’s not time that’s wasted — it’s time that’s spent by the kernel doing useful stuff (as long as you’re not thrashing, so look at the number ...


11

You can't spawn threads from a shell. You don't want to write to the same file from multiple processes. If all your random program does is generate a single number, it should be fast enough that your loop will be io bound. if you can, you should edit it to take an argument and print that many numbers. if the actual execution is the bottleneck, you ...


11

I do use GNU parallel for that, most specifically you can use this recipe: parallel --tag --nonall --slf your.txt command With your.txt being the file with the server IP address/names.


11

tmux has this capability. (along with many other useful capabilities in the same vein) Can be done via: :setw synchronize-panes on


10

#!/bin/bash # set -x # debug version N=${1:-123} n=${2:-45} workers=${workers:-${3:-10}} ((workers < 1)) && ((workers = 1)) ((workers > 20)) && ((workers = 20)) ((min=100000000000000)) #set min to some garbage value work() { for i in ${*}; do for (( j=1; j<=${n}; j++ )); do val=$(/path/to/a.out) val2=$(echo ${val}...


10

GNU parallel is made for just this sort of thing. You can run your script many times at once, with different data from your input piped in for each one: cat input.txt | parallel --pipe your-script.sh By default it will spawn processes according to the number of processors on your system, but you can customise that with -j N. A particularly neat trick is ...


10

You can use a for loop to loop over the individual files and then use tee combined with process substitution (works in Bash and Zsh among others) to pipe to different checksummers. Example: for file in *.mkv; do tee < "$file" >(sha256sum) | md5sum done You can also use more than two checksummers: for file in *.mkv; do tee < "$file" >(...


9

Multixterm Another tool to add to the list is one called multixterm. It uses xterm terminals. You can invoke it like so: $ multixterm And once up you'll be presented with a GUI.                              &...


9

It can be achieved using iTerm2 Reference: http://korishev.com/blog/2014/02/28/iterm2-broadcast-input/


Only top voted, non community-wiki answers of a minimum length are eligible