Questions tagged [slurm]
SLURM is a workload manager for Linux clusters
62
questions
0
votes
0
answers
81
views
Contradiction in GPU numbering of `$CUDA_VISIBLE_DEVICES` and `nvidia-smi` when MIG is active
My aim is very simple. We have multiple GPUs on each node. However, if I allocate only two GPUs for me. nvidia-smi or nvidia-smi -L shows a list of all GPUs including those being used by others and ...
0
votes
0
answers
10
views
Ways to supply common arguments to qsub/slurm using single file?
My team is building a small HPC pipeline that requires multiple cluster jobs to be submitted.
So far, we've been using 5+ submission scripts, but this makes changing configuration options a hassle.
We ...
- 13
0
votes
0
answers
25
views
Small TaskQueue shared on two computers
There are two computers with 12 physical cores each.
Computer A should accept jobs and distribute them among A and B
I want to setup Computers A and B such that
A will accept jobs (via ssh) and ...
- 137
0
votes
0
answers
13
views
Fixing the threads to cores in slurm (and avoid the OS moves them)
I am having problems with with the (infinity) of srun options. Does any one have a recipe
to allocate threads to cores, and more importantly, avoid that there is migration of the threads among the ...
- 1
0
votes
0
answers
59
views
Scheduling difference manual execution vs Slurm srun/sbatch
tl;dr A CPU intensive workload ignores the nice values of other processes only when started via Slurm sbatch/srun (not when manually started).
I am running a systemd service which periodically gathers ...
- 1
0
votes
0
answers
14
views
Controlling HPC Slurm job dependencies by list of filenames
I am quite new to slurm and I am trying to set up a job dependencies pipeline using a master script called dependencies.sh, which contains something like:
jid1=$(sbatch step1.sh)
jid2=$(sbatch --...
- 101
1
vote
1
answer
30
views
Running jobs on HPC cluster using SLURM
Suppose I have two Python scripts: test1.py and test2.py
How do I write a SLURM script to run these files on two different nodes simultaneously?
Note that: test1.py and test2.py are independent of ...
- 11
0
votes
0
answers
42
views
Bash process gets killed in HPC
I am new to bash scripting and I am using the below script to automate my job submission.
This scripts waits for the previous job to finish and automatically submits a new job.
while true
do
...
- 107
3
votes
2
answers
252
views
Passing Argument to Comment in .sh script
I am a beginner in the use of .sh scripts so please excuse my ignorance. This is my problem:
To submit my jobs to our cluster the corresponding submit file has to contain a "slurm header" ...
- 133
0
votes
1
answer
82
views
list out all computer node connected to master node
I have HPC access, connected to other HPC via local IP or just simple hostname (i.e., compute1 or node3). I can log in to them by typing "ssh node3" or its IP like "ssh 10.2.6.20"
...
- 63
1
vote
1
answer
40
views
Tools for batching N commands over L scripts (for N≫L)?
Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to M jobs either running or pending at any one time, of ...
- 13k
0
votes
1
answer
58
views
cant pipe SLURM `module list`command
I am working on a SLURM cluster and there is a command to list all loaded software modules. I want to process the output and i.e. grep it for a certain word. However if I try to use the pipe I get ...
- 165
1
vote
1
answer
91
views
simulating user input in a bash script on slurm
I believe my problem is similar as found here but I was confused by the reply (and I can't comment in this question yet due to my low rep).
I have a bash script I need to run using slurm, so when the ...
0
votes
0
answers
16
views
slurm: CPUs=8 versus Sockets=1 CoresPerSocket=4 ThreadsPerCore=2
i switched to the slurm batch system recently. i have a laptop with 1 socket, 4 cores, and 2 threads per core. when i set up slurm using explicity socket, core and thread info, i canNOT run 8 ...
- 11
0
votes
0
answers
73
views
SLURM: "unsafe" code, forking and restarting processes which exceed memory limit
I am running code in the open-source computer algebra system Sage in a SLURM-managed multi-node cluster environment. The code is highly parallel: I use the Python package mpi4py to manage a task queue ...
1
vote
0
answers
57
views
Slurm multifactor priority - how to configure Slurm to consider past usage?
I'm trying to configure our Slurm to allocate users' jobs according to past usage.
The goal is to have the job priority consider past time usage of mainly GPUs.
I understand I must use the FairShare ...
- 11
0
votes
0
answers
24
views
Slurm 18 allocates MPI jobs to the same CPUs on the node
For some reason when launching an mpi job with SLURM on CentOS8 cluster, slurm ties mpi processes to CPUs always starting from CPU0.
Say there are 128 CPU cores on a compute node. I launch mpi job ...
- 81
1
vote
1
answer
347
views
Python error only when I run script on Linux cluster: _tkinter.TclError: no display name and no $DISPLAY environment variable
My question is related to a python error, but I suspect that it is more a Linux question than a python one. Thus I post it first here.
I am running a python script which does a calculation and then ...
- 155
0
votes
1
answer
310
views
How do I print slurm variables to standard slurm output?
I want to print the following variables to the standard output file slurm-XXXXX.out produced by Slurm. Right now, I am generating a separate .info file for every job.
echo "SLURM Job ID : $...
- 103
1
vote
1
answer
33
views
Very new to slurm. How to get slurm to run multiple core jobs on my linux cluster?
I've been trying to move some existing processes to a revamped linux cluster that now runs on slurm. I thought I have it done, but my problem now is trying to get multiple cores to run.
Here is my ...
- 11
1
vote
1
answer
364
views
Why it's not possible to run wget with background option in slurm script?
I used this script for downloading files. Without -b, wget download files one by one. With -b, I have the possibility to download files in background but also simultaneously. Unfortunately, the script ...
- 11
1
vote
1
answer
175
views
How to use Active Directory (domain groups) for user management in SLURM?
We have a GPU cluster (centOS 8) in our company network which runs SLURM.
Now we would like to realize the SLURM user management via our windows domain controller (active directory).
Every user which ...
- 111
0
votes
1
answer
96
views
SLURM: How do I submit multilple OpenMP parallel codes in a job script
I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.)
#!/bin/bash
#SBATCH -N 1
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=2
export OMP_NUM_THREADS=...
- 103
0
votes
1
answer
302
views
How to save the output of slurm command into a file?
I have installed slurm command to track the amount of data consumed by an interface apt install slurm
My gold is to save its output into a file, I wrote this script to let me do that, but I did not ...
- 3
0
votes
1
answer
108
views
Canceling SLURM jobs without receiving email notifications
Given some SLURM jobs which were started using sbatch --mail-type=ALL, is there any way to cancel them without receiving the email notifications informing you that they were canceled? I was thinking ...
- 282
1
vote
2
answers
2k
views
Passing configure option to rpmbuild?
To build a piece of software I normally do
rpmbuild -ta slurm*.tar.bz2
However I now need to configure the software with the option
--with-pmix=/home/user/git/pmix/install/2.1
Is this possible ...
- 123
1
vote
0
answers
73
views
Is it possible to run users' jobs in chroot environment using Slurm
I'm using Slurm. I want users only to access their own /home/ directory (or directory that they executed their job), ubuntu packages that is installed by the sudo user, and binary files user /bin, ...
- 229
2
votes
2
answers
2k
views
"#" (comments) is SLURM job submissions
The question is related to the article Introducing SLURM by Princeton Research Computing.
For instance,
#SBATCH --job-name=slurm-test # create a short name for your job
SBATCH after the first # ...
- 123
1
vote
1
answer
58
views
Run a shell script within an already running shell script
I have a small SLURM cluster, which I am using to run jobs. I run a script and while that script is running, I want to enter a specific key command "t + enter". I believe I should do it by submitting ...
- 111
2
votes
1
answer
731
views
Slurm on desktop computers, how to prioritize the owner
In short: On a Slurm cluster, I need some computers to be available and responsive to their respective owners during work hours.
Problem: I manage a small (but growing) heterogeneous cluster with ...
- 141
0
votes
0
answers
477
views
SLURM partition configuration: suspend and requeue
I am struggling a bit to configure our SLURM server. I wanted to have several queues (partitions) so that the short queues had higher priority and jobs could enter immediately suspending or requeuing ...
- 1
0
votes
2
answers
720
views
Does the "watch" command put stress on a scheduler?
I have an account on a compute cluster which uses the SLURM scheduler. I have some jobs in the queue and I'm using the "watch" command to see their status:
watch squeue -u myUserName
Does constantly ...
0
votes
1
answer
2k
views
How to execute a script on several nodes using Slurm?
I have a R code that I want to execute on several nodes using Slurm, with each iteration of my paramater which goes on a node. This is my Slurm code :
#!/bin/bash
#SBATCH -o job-%A_task.out
#SBATCH -...
- 101
1
vote
0
answers
81
views
How to get CPU "load" of subset of CPU cores?
In my environment, each job is assigned (cpuset) to a subset of CPU cores, almost exclusively. (Actually, it's done by Slurm.)
Since the CPU core occupation is exclusive, can I get/calculate the 1-...
- 274
3
votes
1
answer
243
views
restore $0 or $BASH_SOURCE after it is modified by the cluster
I am using a shared SLURM cluster. I am trying to get the path of the bash script from inside the script itself.
There is already an excellent thread here: https://stackoverflow.com/questions/59895/...
- 199
1
vote
1
answer
764
views
SLURM: How to determine maximum --cpus-per-task and --mem-per-cpu?
How can I determine the optimum/maximum number of CPUs per task when running a job? Is there a way to display the total available memory on a given CPU as well?
- 153
0
votes
1
answer
690
views
Maximum timeout for process
I have a script which I think can take few days to finish. Basically it submits slurm jobs when queue isn't full and in the process it can wait a while.
The script results in incomplete results, ...
- 101
2
votes
0
answers
456
views
Running a slurm job array [closed]
I have a job_array.sh with 23 tasks:
> angsd -i ./realigned_RSFV1A.bam -anc ./BF_genome.fa -gl 1 -doSaf 1 -doSnpStat 1 -doMajorMinor 1 -doMaf 1 -doGeno 6 -doCounts 1 -doPost 1 -nThreads 24 -out ./...
1
vote
2
answers
1k
views
How to list all the processes assigned to a user on a cluster
We have a cluster in our university controlled by SLURM. The issue is that I have observed sometimes some of the resources are not being realised even though they do not show up in the squeue command. ...
- 379
4
votes
2
answers
1k
views
`watch` command with piping `|` [duplicate]
I want to keep monitoring a specific job on a slurm worload like cluster. I tried to use the watch command and grep the specific id. If the job id is4138, I tried
$> watch squeue -u mnyber004 | ...
- 257
3
votes
1
answer
1k
views
Watch-command-alias-expansion AND need to use quotes
My questions is similar to the watch question here but with a twist. I need to use quotes, which seem to be stripped by an aliased watch.
I want run watch on a custom slurm squeue command:
$alias ...
- 133
21
votes
1
answer
36k
views
How to submit a job to a specific node using Slurm's sbatch command?
Our nodes are named node001 ... node0xx in our cluster. I wonder, is it possible to submit a job to a specific node using Slurm's sbatch command? If so, can someone post an example code for that?
- 659
3
votes
0
answers
291
views
SLURM: restrict GPU access only to SLURM
I have a single machine (Ubuntu 16.04 Server) with
4 TitanX GPUs. This will be a lab machine on which
students will learn about CUDA and stuff. I installed
SLURM because I want a tool to schedule and ...
- 1,171
0
votes
1
answer
2k
views
How to use a variable with SLURM sbatch to set the output/error file name?
We've just switched to using SLURM and I would like to submit a series of jobs using a loop and sbatch. Previously, I could use a variable as part of the output file names. I've been trying to do this ...
- 1
12
votes
2
answers
12k
views
How to cancel jobs on Slurm with job ID(job number) bigger than a certain number?
I have submitted 800 jobs on Slurm. I want to cancel those jobs that have job ID/number bigger than a number(since there is a mistake in them). I don't want to cancel all my jobs because some are ...
- 131
0
votes
1
answer
663
views
launch singularity in a slurm script
I have singularity installed on /usr/local/bin and want to launch singularity from my slurm scirpt on my linux cluster but I keep getting this error "error:/var/spool/slurm/d/job01485/slurm_script: ...
- 21
23
votes
5
answers
24k
views
Best way to cancel all the SLURM jobs from shell command output
I submitted lots of SLURM job script with debug time limit (I forgot to change the time for actual run). Now they are all submitted at the same time, so they all start with job ID 197xxxxx. Now, I can ...
- 382
1
vote
0
answers
467
views
How to increase memory limit and number of cores that Rstudio uses on a cluster
I launched Rstudio using FastX on Rivanna cluster (CentOS Linux release 7.3.1611 (Core)):
https://arcs.virginia.edu/rivanna
The main goal for doing that was to significantly increase the speed of ...
- 121
0
votes
1
answer
1k
views
slurm: srun & sbatch different performance with the same settings
In a slurm system, when I use srun command to run the program. It runs very slow and seems like only one processor works.
srun --pty -A free -J test -N 1 -n 1 -c 1 mpirun -np 16
$FEAPHOME8_3/...
- 23
2
votes
1
answer
11k
views
Automatically input "yes" on the bash file [closed]
Slurm is workload manager. There are two kinds of modes to run job, interactive(srun) and batch mode(sbatch).
When using interactive mode, one needs to leave the terminal open which may lead extra ...
- 23