Questions tagged [multithreading]
The multithreading tag has no usage guidance.
145
questions
0
votes
1
answer
39
views
Linux multiprocessors support
I am reading the following paper. In the paper, the authors argue that Unix/Linux "has struggled for a decade to support multiprocessors in a single node" in the last paragraph of the first ...
- 3
0
votes
1
answer
37
views
How can I create a dummy multi-threaded process and get its tids using common utilities?
I can make a dummy process that will hang forever and get its PID with:
bash -c 'echo $$; read'
I'd like a multi-threaded alternative to this one-liner which creates 2+ threads, prints the thread ids ...
- 147
0
votes
0
answers
20
views
user and kernel threads mapping
In this link in the many to many model , it states that :
The user thread programming facilities are shared between the threads
library and the kernel threads.
What does this exactly mean?
- 87
1
vote
0
answers
23
views
File locking from different therads of the same process, including for Solaris OS
I need to perform a file lock when accessing a file from C/C++ code with the following requirements:
Blocking should be performed when access occurs not only from different processes, but also from ...
- 205
1
vote
1
answer
112
views
How to run one command on several cores
I have a command which I want to run on all free cores to speed up the execution time. Specifically I am running the Pitman-Yor Adaptor-Grammar Sampler software I downloaded from here
./py-cfg/py-cfg-...
- 167
1
vote
1
answer
49
views
Understanding load average on multicore system with a multithreaded app
We have a curious situation regarding load average on our system. It's running an application called ZAG that is idle most of the day. But every 80 minutes or so, it has some sort of activity burst of ...
- 2,249
1
vote
2
answers
128
views
Spike in number of page faults with make -j`nproc`
I am running a benchmark to figure out the number of jobs I should allow GNU Make to use in order to have optimal compile time. To do so, I am compiling Glibc with make -j<N> with N an integer ...
1
vote
1
answer
213
views
Understanding overhead cost of context switching
I've always read/heard that context switching is expensive. I recently started reading Robert Love's "Linux Kernel Development" and have made it through the Processes and Process Scheduling ...
- 111
0
votes
0
answers
49
views
When to use named FIFO over posix message queues for IPC and process-thread communication
Trying to understand which IPC mechanism (mqueue vs mkfifo) is preferred to use in which scenario.
I have got two different processes/tasks and multiple threads running in one of the processes. The ...
- 21
1
vote
1
answer
197
views
Linux equivalent of Thread Director?
The 12th Generation Intel chips are rumoured to have "Efficient" and "Performance" cores. Apparently Microsoft's Thread Director, in Windows 11, takes advantage of them.
What will ...
- 111
0
votes
1
answer
131
views
Help understanding lsof and threads
When i use -p $PID on lsof i get only the immediate thread of the process. why and how to work with threads and lsof effectively?
sudo lsof -p 5344 > lsof-p5344.log
sudo lsof | grep '\<5344\>...
- 1
1
vote
1
answer
284
views
Why do threads have their own PID?
I'm using htop and looking at a process (rg) which launched multiple threads to search for text in files, here's the tree view in htop:
PID Command
1019 |- rg 'search this'
1021 |- rg 'search ...
- 304
1
vote
1
answer
215
views
xz: OOM when compressing 1TB .tar
I'm trying to compress a large archive with multi-threading enabled, however, my system keeps freezing up and runs out of memory.
OS: Manjaro 21.1.0 Pahvo
Kernel: x86_64 Linux 5.13.1-3-MANJARO
...
- 113
0
votes
0
answers
40
views
Multithreading Prgram in C++
I am running the following multi-threaded C++ program:
#include <bits/stdc++.h>
#include <algorithm>
#include <ctime>
#include <ratio>
#include <chrono>
#include <...
2
votes
2
answers
148
views
Simple BASH script to become a multi-process/'threaded' BASH script?
I have the following, working BASH script, which I run on a Mac Pro 2010/Mojarve OS:
#!/bin/bash
c=0
cnt=0
# count up wav files
cnt=$(find /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/ -name "*....
0
votes
0
answers
42
views
How to create a thread which does not undergo context switch during the execution?
I am have multi process unix system where each user space process have multiple threads. Here I am facing an issue with one of the thread as the thread get switched in between and when it comes back ...
3
votes
3
answers
376
views
How can I parallelize command sha256sum or other hashing commands?
I want to parallelize hash calculation process because I have a very large amount of file counts and sizes.
When I see CPU usage of these commands, I get upset because they are only using one thread; ...
- 31
1
vote
1
answer
84
views
FIFO number of processes, sustained growth
I have a FIFO question:
Why the fifo thread_num limit 2,
but number of processes, sustained growth when one process ends,
two processes start, and then got 3 processes.
sustained growth 2n-1
#!bin/...
- 11
2
votes
2
answers
577
views
How to run a bash script using threads/parallel
I am running the following bash script as is. I.e. sequentially. Note that the prefetch and fastq-dump are calls for downloading files from a database, and _shrink is a C++ program that I have made to ...
0
votes
1
answer
298
views
Which thread receives SIGURG?
I recently wrote a "study note" about Unix, and I made following proposition about multi-threaded processes:
it will be almost impossible for the kernel to identify the thread that should ...
- 429
3
votes
3
answers
267
views
Boosting the grep search using GNU parallel
I am using the following grep script to output all the unmatched patterns:
grep -oFf patterns.txt large_strings.txt | grep -vFf - patterns.txt > unmatched_patterns.txt
patterns file contains the ...
- 71
2
votes
0
answers
50
views
JVM only uses 4 cores
I have a processor-intensive Java program and I want it to use all 8 available cores on the machine.
I've run taskset and it reports the mask to be:
pid 5897's current affinity mask: ff
But in top (...
- 121
0
votes
1
answer
152
views
How to switch from one process to another process and kill the first process
I have the following scenario, I have two programs running one in the background and one in front. The back program is doing some stuff for the front program. once the back program has done the ...
0
votes
1
answer
56
views
Bind 9 process of named is using one core cpu only
i Have a DNS server running on Debian os , the named process is taking 394% of cpu.
i noticed it is using one core of the 1 cores provided to the whole VM .
top - 17:29:04 up 105 days, 1:33, 1 user, ...
- 118
0
votes
1
answer
95
views
Schedule jobs from a queue onto multiple threads
I have a function that has to process all files in a set of directories (anything between 5-300 files). The number of parallel threads to be used is user-specified (usually 4). The idea is to start ...
- 135
2
votes
1
answer
71
views
All processes running on the same core
I found all processes on my machine to only run on a single core and their core affinity set to 0.
Here is a small python script which reproduces this for me:
import multiprocessing
import numpy as np
...
- 21
13
votes
3
answers
14k
views
How to use multi-threading for creating and extracting tar.xz
I use
tar -cJvf resultfile.tar.xz files_to_compress
to create tar.xz and
tar -xzvf resultfile.tar.xz
to extract the archive in current directory. How to use multi threading in both cases? I don't ...
- 237
-1
votes
1
answer
689
views
How to find out on which core a thread is running on?
Let's say we have a CPU-intensive application called multi-threaded-application.out that is running on top of Ubuntu with a PID of 10000. It has 4 threads with tid 10001, 10002, 10003, and 10004. I ...
3
votes
2
answers
195
views
What are these threads named disk$0, disk$1, etc.?
I am using Ubuntu 20.04 LTS. The kernel version is 5.4.0-42.
Here is an example program:
// mre.c
// Compile with: cc -o mre mre.c -lSDL2
#include <stdio.h>
#include <SDL2/SDL.h>
int main(...
- 133
0
votes
1
answer
136
views
Does a process filling all logical cores have a negative impact on performance? [closed]
Because of hyper-threading, my CPU has 2 logical processors per core. If I understand the premise of hyper-threading correctly, it allows each core to have a separate cache and instruction pointer for ...
- 131
0
votes
2
answers
252
views
massive resource consumption during mkfs.ntfs on a zvol, why (and how can I limit this)?
On my machine mkfs.ntfs is slow and results in massive use of resources, preventing me from using the machine for anything else. According to top it (or rather directly related zvol processes) is ...
- 629
18
votes
4
answers
3k
views
Are basic POSIX utilities parallelized?
In a common Linux distribution, do utilities like rm, mv, ls, grep, wc, etc. run in parallel on their arguments?
In other words, if I grep a huge file on a 32-threaded CPU, will it go faster than on ...
- 361
0
votes
0
answers
477
views
IPSec multithreading via pcrypt worse than single thread
According to this document:
https://www.strongswan.org/docs/Steffen_Klassert_Parallelizing_IPsec.pdf parallelizing IPSec using pcrypt kernel module will make the connection much more efficient (graphs ...
- 11
1
vote
1
answer
343
views
Threads to process a million files [closed]
How many threads should be used to process a million files? How yould you justify your answer? This is a question from an OS exam from last year and I'm courious how you guys think. I think that 10....
- 15
1
vote
1
answer
357
views
In Solaris box how do we check if hyper-threading/multi-threading is enabled or not using command?
In Solaris, how we can find the status or check hyper-threading or multi-threading is enabled or not using commands?
- 11
0
votes
0
answers
325
views
macOSx / what is the kernel variable os_refcnt counting and what might be causing it to overflow?
Trying to figure out why running some software that I am developing causes my Mac to do a panic restart every 16 h or so always with an "os_refcnt" kernel variable overflow when running tcsh scripts.
...
- 21
23
votes
1
answer
3k
views
How to use "parallel" to speed up "sort" for big files fitting in RAM?
I have a 100 M line file that fits in RAM on a GNU/Linux system.
This is rather slow:
sort bigfile > bigfile.sorted
and does not use all 48 cores on my machine.
How do I sort that file fast?
- 29.6k
0
votes
1
answer
84
views
Thread safe next number (or round-robin) service in linux
I have an Ubuntu machine and a scenario in which I need to generate the next number (starting from X) between 0 and 20 in a round-robin way.
I need a way to get the next number which will be thread ...
- 101
1
vote
0
answers
197
views
Using libhugetlbfs to transparently back up glibc malloc calls in a multi-threaded application
I'm trying to back up the memory allocations of a multi-threaded application with 1GiB hugepages using libhugetlbfs. However, only the main thread allocations are being assigned hugepages. If I ...
- 295
0
votes
1
answer
2k
views
What types of threads does Java/JVM use from Linux OS's perspective?
A developer friend of mine recently asked the question: On a Linux system when a Java application runs which has threads, how do these threads appear to the underlying Linux OS?
So what are Java ...
slm♦
- 344k
3
votes
1
answer
7k
views
Multithreaded xz, with gzip, pv, and pipes - is this the most efficient I can get?
I'm excited to learn that xz now supports multithreading:
xz --threads=0
But now I want to utilise this as much as possible. For example, to recompress gzips as xz:
gzip -d -k -c myfile.gz | pv | ...
3
votes
1
answer
417
views
Why is this code exiting with return code 16?
I am trying my hands on clone() system call to create a Thread. However program is terminating itself as it return from t2_thread() function. Why is this behaviour? What am I missing?
#define ...
- 67
1
vote
1
answer
692
views
How to correctly create and stop kthreads?
I wrote the following kernel module:
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/delay.h>
#include <linux/kthread.h>
#include <...
0
votes
1
answer
76
views
How to limit the amount of spillover to other CPUs from a single process
Please excuse the complete ignorance of my question - I'm probably lacking the right terminology here...
I am trying to understand how a single process (R session) can spill over to all CPUs on our ...
- 101
2
votes
0
answers
134
views
Linux task state different in /proc/pid/stat compared to Ftrace output of sched events
I'm trying to trace the flow of a piece of code just to understand the points where the tasks go into D/S/R states (uninterruptible/interruptible/running). I do understand the meaning of each state, ...
- 41
0
votes
2
answers
342
views
Lightweight processes behavior with an new PID
I've been experimenting with lightweight processes. Basically calling the clone function and assigning a new PID to the cloned LWP. This works fine it lets me identify all the children threads of ...
- 111
1
vote
0
answers
64
views
Behavior of Linux kernel with respect to machine check exceptions (MCE) and threads
I was recently working with custom hardware that was not fully respecting the PCIe bus standards, and certain reads from the device would cause a bus fault and Linux machine check exception. When this ...
- 11
3
votes
1
answer
330
views
Is mkdir -p safe to parallelize?
I'm writing Makefile rules like this:
out/%.x: ...
mkdir -p $(@D)/
...
I intend to parallelize this Makefile (-j), which will trigger many mkdir -p's to simultaneously.
Is this safe? Or do ...
- 2,873
4
votes
0
answers
499
views
Atomic write and tell for files in O_APPEND mode on linux
I am dealing with a situation where I want to save state persistently for an application that has many processes and many threads within those processes. Normally this would require a heavy use of ...
- 171
5
votes
1
answer
6k
views
'pthread_create: Resource temporarily unavailable' when running multiple docker instances
TLDR
When spinning up multiple docker containers in which I run npm ci, I start getting pthread_create: Resource temporarily unavailable errors (less than 5 docker containers can run fine). I deduce ...
- 2,940