80

I'm not entirely sure what you're asking here. Yes, top shows CPU usage as a percentage of a single CPU by default. That's why you can have percentages that are >100. On a system with 4 cores, you can see up to 400% CPU usage. You can change this behavior by pressing I (that's Shift + i and toggles "Irix mode") while top is running. That will cause it to ...


60

The common protocols HTTP, FTP and SFTP support range requests, so you can request part of a file. Note that this also requires server support, so it might or might not work in practice. You can use curl and the -r or --range option to specify the range and eventually just catting the files together. Example: curl -r 0-104857600 -o distro1.iso '...


54

As mavillan already suggested, just use terminator. It allows to display many terminals in a tiled way. When enabling the broadcasting feature by clicking on the grid icon (top-left) and choosing "Broadcast All", you can enter the very same command simultaneously on each terminal. Here is an example with the date command broadcasted to a grid of 32 ...


49

nproc gives the number of CPU cores/threads available, e.g. 8 on a quad-core CPU supporting two-way SMT. The number of jobs you can run in parallel with make using the -j option depends on a number of factors: the amount of available memory the amount of memory used by each make job the extent to which make jobs are I/O- or CPU-bound make -j$(nproc) is a ...


34

There are two easy solutions for this. Basically, using xargs or parallel. xargs Approach: You can use xargs with find as follows: find . -type f -print0 | xargs -0 -P number_of_processes grep mypattern > output Where you will replace number_of_processes by the maximum number of processes you want to be launched. However, this is not guaranteed to ...


29

Check out pee ("tee standard input to pipes") from moreutils. This is basically equivalent to Marco's tee command, but a little simpler to type. $ echo foo | pee md5sum sha256sum d3b07384d113edec49eaa6238ad5ff00 - b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c - $ pee md5sum sha256sum <foo.iso f109ffd6612e36e0fc1597eda65e9cf0 - ...


28

Use wait. For example: Data1 ... > Data1Res.csv & Data2 ... > Data2Res.csv & wait AnalysisProg will: run the Data1 and Data2 pipes as background jobs wait for them both to finish run AnalysisProg. See, e.g., this question.


27

On linux, the system call to set the CPU affinity for a process is sched_setaffinity. Then there's the taskset tool to do it on the command line. To have that single program run on only one CPU, I think you'd want something like taskset -c 1 ./myprogram (set any CPU number as an argument to the -c switch.) That should be close enough to a single-...


27

You can get a first impression by checking whether the utility is linked with the pthread library. Any dynamically linked program that uses OS threads should use the pthread library. ldd /bin/grep | grep -F libpthread.so So for example on Ubuntu: for x in $(dpkg -L coreutils grep findutils util-linux | grep /bin/); do if ldd $x | grep -q -F libpthread.so; ...


24

This looks like a job for gnu parallel: parallel bash -c ::: script_* The advantage is that you don't have to group your scripts by cores, parallel will do that for you. Of course, if you don't want to babysit the SSH session while the scripts are running, you should use nohup or screen


22

Confiq's answer is a good one for small i and j. However, given the size of i and j in your question, you will likely want to limit the overall number of processes spawned. You can do this with the parallel command or some versions of xargs. For example, using an xargs that supports the -P flag you could parallelize your inner loop as follows: for i in {...


20

Yes, it is. If you want to do two things concurrently, and wait for them both to complete, you can do something like: sh ./stay/get_it_ios.sh & PIDIOS=$! sh ./stay/get_it_mix.sh & PIDMIX=$! wait $PIDIOS wait $PIDMIX Your script will then run both scripts in parallel, and wait for both scripts to complete before continuing.


20

If you're using GNU xargs, there's --process-slot-var: --process-slot-var=environment-variable-name Set the environment variable environment-variable-name to a unique value in each running child process. Each value is a decimal integer. Values are reused once child processes exit. This can be used in a rudimentary load distribution scheme, for ...


17

tmux has this capability. (along with many other useful capabilities in the same vein) Can be done via: :setw synchronize-panes on


17

for((i=1;i<100;i++)); do nohup bash script${i}.sh & done


16

A problem with split --filter is that the output can be mixed up, so you get half a line from process 1 followed by half a line from process 2. GNU Parallel guarantees there will be no mixup. So assume you want to do: A | B | C But that B is terribly slow, and thus you want to parallelize that. Then you can do: A | parallel --pipe B | C GNU Parallel by ...


16

$! is guaranteed to give you the pid of the process in which the shell executed that tail command. Shells are single threaded, each shell lives in its own process with its own set of variables. There's no way the $! of one shell is going to leak into another shell, just like assigning a shell variable in one shell is not going to affect the variable of the ...


15

A better way would be to use GNU Parallel. GNU parallel is simple and with it we can control the number of jobs to run in parallel with more control over the jobs. In the below command, script{1..3}.sh gets expanded and are sent as arguments to bash in parallel. Here -j0 indicates that as many jobs should be run as possible. By default parallel runs one job ...


13

When you write A | B, both processes already run in parallel. If you see them as using only one core, that's probably because either of CPU affinity settings (perhaps there is some tool to spawn a process with different affinity) or because one process isn't enough to hold a whole core, and the system "prefers" not to spread out computing. To run several B'...


12

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel: ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > {}res....


12

The most straight-foward way is to use nproc like so: make -j`nproc` The command nproc will return the number of cores on your machine. By wrapping it in the ticks, the nproc command will execute first, return a number and that number will be passed into make. You may have some anecdotal experience where doing core-count + 1 results in faster compile ...


12

Red represents the time spent in the kernel, typically processing system calls on behalf of processes. This includes time spent on I/O. There’s no point in trying to reduce it just for the sake of reducing it, because it’s not time that’s wasted — it’s time that’s spent by the kernel doing useful stuff (as long as you’re not thrashing, so look at the number ...


11

I do use GNU parallel for that, most specifically you can use this recipe: parallel --tag --nonall --slf your.txt command With your.txt being the file with the server IP address/names.


11

lftp would do this with the command mirror -R -P 20 localpath - mirror syncs between locations, and -R uses the remote server as the destination , with P doing 20 parallel transfers at once. As explained in man lftp: mirror [OPTS] [source [target]] Mirror specified source directory to local target directory. If target directory ends with a ...


11

There are at least two ways of speeding up grep CPU-wise: If you're searching for a fixed string rather than a regular expression, specify the -F flag; If your pattern is ASCII-only, use an 8-bit locale instead of UTF-8, e.g. LC_ALL=C grep .... These won't help though if your hard drive is the bottleneck; in that case probably parallelizing won't help ...


10

It can be achieved using iTerm2 Reference: http://korishev.com/blog/2014/02/28/iterm2-broadcast-input/


10

GNU parallel is made for just this sort of thing. You can run your script many times at once, with different data from your input piped in for each one: cat input.txt | parallel --pipe your-script.sh By default it will spawn processes according to the number of processors on your system, but you can customise that with -j N. A particularly neat trick is ...


10

You can use a for loop to loop over the individual files and then use tee combined with process substitution (works in Bash and Zsh among others) to pipe to different checksummers. Example: for file in *.mkv; do tee < "$file" >(sha256sum) | md5sum done You can also use more than two checksummers: for file in *.mkv; do tee < "$file" >(...


9

Very basic setup: for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done Authenticating with name/password is really no good idea. You should set up a private key for this: ssh-keygen && for host in $(cat hosts.txt); do ssh-copy-id $host; done


9

Multixterm Another tool to add to the list is one called multixterm. It uses xterm terminals. You can invoke it like so: $ multixterm And once up you'll be presented with a GUI.                              &...


Only top voted, non community-wiki answers of a minimum length are eligible