Questions tagged [slurm]
SLURM is a workload manager for Linux clusters
78
questions
0
votes
0
answers
27
views
Looking for the Nvida devices in my WSL2 Ubuntu implementation for SLURM GRES
I am trying something odd. I have a slurm cluster configured that has 4 compute nodes. 2 of which are Windows 11 machines running WSL2 and I have it working.
right now, I am trying to add GPU support ...
0
votes
0
answers
31
views
Bash script ignores all terminal input following an underscore [duplicate]
Goal
My server manages jobs using SLURM. Each job in the queue has a job ID, which can be either a single integer or two integers separated by an underscore, e.g., 123_4. You can use scontrol show job ...
- 1
0
votes
0
answers
20
views
Slurm not preserving QoS limits
I am trying to set QoS limits on the cluster,
to be specific, I want to make sure nobody is able to submit jobs to a specific partition.
So I enable CPU limits on the partition and it works as ...
- 93
0
votes
0
answers
16
views
Changing SSH keys on the compute node in pcluster using slurm
Our login-node is currently one of the compute node, residing in a separate queue. We want to setup custom SSH keys on /etc/ssh for that login-node.
We tried pulling in the custom SSH keys on the post-...
- 93
0
votes
1
answer
49
views
Naming scheme for compute nodes in a slurm cluster?
I am planning to build a new computer cluster based on Debian 11, Slurm, and a diverse range of hardware. The cluster will boot from a network location using PXE. It is expected that the cluster will ...
- 591
0
votes
1
answer
57
views
Slurm is making my python script wait in running status
When I submit the following Slurm file with sbatch, the job has an immediately running status. However, the Python script runs after about 10 minutes. Also, I tried the same with srun command, and ...
- 1
1
vote
0
answers
75
views
How to access `/scratch` directory of nodes 2 to N in SLURM when node 1 doesn't have access to it?
I'm running a SLURM job with several nodes, where the nodes are arranged in the order of 1, 2, 3, 4, 5 ... N. However, I'm facing an issue where the node 1 doesn't have access to the /scratch ...
- 109
0
votes
0
answers
44
views
Use more CPU cores using code in PyTorch that also uses GPU
I'm trying to successfully run code in PyTorch that uses DataLoader. It is possible to configure the DataLoader to load data using several processes (which speeds up data loading a lot), via the use ...
- 1
0
votes
0
answers
319
views
Using a node by its IP address
I was trying to use a specific node by its IP address during a slurm job. I know we can specify the node inside the .sh file by using the
#SBATCH --nodelist =ncbxxx
command where we specify the names ...
2
votes
0
answers
145
views
How does SLURM authentication and encryption work?
I'm looking into the security of SLURM. None of the documentation is very clear about how it works. So far I have ascertained that authentication of users is done using MUNGE.
As far as I can tell ...
- 537
1
vote
1
answer
792
views
Running multiple SLURM jobs on the same GPU
So, I am by no means a sysadmin but I need to use an existing SLURM installation to launch a sizable amount of jobs (around 5000).
The cluster is composed of 1 node with 10 GPUs (with 8GB of memory ...
- 113
1
vote
1
answer
268
views
slurm: one compute nodes not in sinfo?
I am setting up slurm 22.05.6, slowly building a cluster. So far I have set up one server, vogon, and a node, ceres; this seem to work fine - I can start jobs with srun. The server is on Debian 11, ...
- 591
-2
votes
1
answer
193
views
How to display only the number of jobs running in HPC by specific user?
I want to Display only the number of jobs running in the HPC related to my username!
I don't wanna see all the jobs as the command squeue does!
Thanks!
- 1
0
votes
0
answers
33
views
Launch Program in Background from a Epilog Script
I want to start a program in the background from a epilog script. I put the following content into the epilog script:
#! /bin/bash
/usr/bin/nohup /path/to/my/program &
I know the script is being ...
5
votes
1
answer
416
views
Is munge required for a single node slurm setup?
I'm installing slurm on a single server to be used for scheduling purposes among a small group of people. There is not now, nor will there ever be, an intent to scale beyond this single node.
Is munge ...
- 1,905
1
vote
1
answer
343
views
run slurm job on already logged on nodes
Is it possible to run a slurm job on already logged on nodes of a cluster?
Suppose I am already logged on nodes:
casade01
casade02
casade03
so that I don't need to wait in a queue. If it matters, I ...
- 131
2
votes
1
answer
148
views
How to split the content of a folder into subfolders
I would like to know if there is an easy way to automatically (programmatically, within a bash script) to split the content of a folder into subfolders with the parent folder.
I have a folder let's ...
- 195
0
votes
1
answer
1k
views
How to find which node name is invalid in Slurm error: "sbatch: error: Batch job submission failed: Invalid node name specified"
I'm getting this error "sbatch: error: Batch job submission failed: Invalid node name specified" for this bash script
#!/bin/bash -l
#SBATCH --gpus=1
#SBATCH -p overcap
#SBATCH -A overcap
#...
- 59
1
vote
1
answer
407
views
Slurm - How to cancel jobs that contain a word in the job_name
How do you cancel jobs in Slurm that contain a word in job name? So for example let's say I do squeue in the terminal and get all my current jobs that are running.
JOBID PARTITION NAME USER ST ...
- 59
2
votes
1
answer
333
views
Cancel slurm job range in zsh
I am able to cancel slurm jobs by typing something like the following:
$ scancel 66421802_[11-20]
On bash, this works fine. However, on zsh I get the following error:
$ scancel 66421802_[11-20]
zsh: ...
- 233
1
vote
0
answers
134
views
How does salloc --exclusive behave on a crowded cluster?
In case of salloc --exclusive, the slurm man page is quite optimistic that it will find a node that is empty enough to grant exclusive access to the submitter.
What happens, though, if this is not the ...
- 113
-1
votes
1
answer
157
views
How to set bash script parameter to file minus the extension
so I have this bash file called test.sh
#!/bin/bash -l
out_file=`basename "$0"`
#SBATCH --gpus=1
#SBATCH -p long
#SBATCH -o err_${out_file::-3}.out
and I want to output the error to ...
- 59
0
votes
0
answers
30
views
Small TaskQueue shared on two computers
There are two computers with 12 physical cores each.
Computer A should accept jobs and distribute them among A and B
I want to setup Computers A and B such that
A will accept jobs (via ssh) and ...
- 197
1
vote
0
answers
147
views
Scheduling difference manual execution vs Slurm srun/sbatch
tl;dr A CPU intensive workload ignores the nice values of other processes only when started via Slurm sbatch/srun (not when manually started).
I am running a systemd service which periodically gathers ...
- 11
1
vote
1
answer
114
views
Running jobs on HPC cluster using SLURM
Suppose I have two Python scripts: test1.py and test2.py
How do I write a SLURM script to run these files on two different nodes simultaneously?
Note that: test1.py and test2.py are independent of ...
- 11
0
votes
0
answers
401
views
Bash process gets killed in HPC
I am new to bash scripting and I am using the below script to automate my job submission.
This scripts waits for the previous job to finish and automatically submits a new job.
while true
do
...
- 117
3
votes
2
answers
3k
views
Passing Argument to Comment in .sh script
I am a beginner in the use of .sh scripts so please excuse my ignorance. This is my problem:
To submit my jobs to our cluster the corresponding submit file has to contain a "slurm header" ...
- 133
0
votes
1
answer
693
views
list out all computer node connected to master node
I have HPC access, connected to other HPC via local IP or just simple hostname (i.e., compute1 or node3). I can log in to them by typing "ssh node3" or its IP like "ssh 10.2.6.20"
...
- 83
1
vote
1
answer
44
views
Tools for batching N commands over L scripts (for N≫L)?
Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to M jobs either running or pending at any one time, of ...
- 14.8k
1
vote
1
answer
237
views
cant pipe SLURM `module list`command
I am working on a SLURM cluster and there is a command to list all loaded software modules. I want to process the output and i.e. grep it for a certain word. However if I try to use the pipe I get ...
- 185
1
vote
1
answer
344
views
simulating user input in a bash script on slurm
I believe my problem is similar as found here but I was confused by the reply (and I can't comment in this question yet due to my low rep).
I have a bash script I need to run using slurm, so when the ...
2
votes
0
answers
152
views
Slurm multifactor priority - how to configure Slurm to consider past usage?
I'm trying to configure our Slurm to allocate users' jobs according to past usage.
The goal is to have the job priority consider past time usage of mainly GPUs.
I understand I must use the FairShare ...
- 21
0
votes
0
answers
60
views
Slurm 18 allocates MPI jobs to the same CPUs on the node
For some reason when launching an mpi job with SLURM on CentOS8 cluster, slurm ties mpi processes to CPUs always starting from CPU0.
Say there are 128 CPU cores on a compute node. I launch mpi job ...
- 71
2
votes
1
answer
656
views
Python error only when I run script on Linux cluster: _tkinter.TclError: no display name and no $DISPLAY environment variable
My question is related to a python error, but I suspect that it is more a Linux question than a python one. Thus I post it first here.
I am running a python script which does a calculation and then ...
- 165
0
votes
1
answer
1k
views
How do I print slurm variables to standard slurm output?
I want to print the following variables to the standard output file slurm-XXXXX.out produced by Slurm. Right now, I am generating a separate .info file for every job.
echo "SLURM Job ID : $...
- 103
2
votes
1
answer
150
views
Very new to slurm. How to get slurm to run multiple core jobs on my linux cluster?
I've been trying to move some existing processes to a revamped linux cluster that now runs on slurm. I thought I have it done, but my problem now is trying to get multiple cores to run.
Here is my ...
- 21
1
vote
1
answer
569
views
Why it's not possible to run wget with background option in slurm script?
I used this script for downloading files. Without -b, wget download files one by one. With -b, I have the possibility to download files in background but also simultaneously. Unfortunately, the script ...
- 11
1
vote
1
answer
421
views
How to use Active Directory (domain groups) for user management in SLURM?
We have a GPU cluster (centOS 8) in our company network which runs SLURM.
Now we would like to realize the SLURM user management via our windows domain controller (active directory).
Every user which ...
- 111
0
votes
1
answer
291
views
SLURM: How do I submit multilple OpenMP parallel codes in a job script
I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.)
#!/bin/bash
#SBATCH -N 1
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=2
export OMP_NUM_THREADS=...
- 103
0
votes
1
answer
1k
views
How to save the output of slurm command into a file?
I have installed slurm command to track the amount of data consumed by an interface apt install slurm
My gold is to save its output into a file, I wrote this script to let me do that, but I did not ...
- 3
0
votes
1
answer
264
views
Canceling SLURM jobs without receiving email notifications
Given some SLURM jobs which were started using sbatch --mail-type=ALL, is there any way to cancel them without receiving the email notifications informing you that they were canceled? I was thinking ...
- 331
1
vote
2
answers
3k
views
Passing configure option to rpmbuild?
To build a piece of software I normally do
rpmbuild -ta slurm*.tar.bz2
However I now need to configure the software with the option
--with-pmix=/home/user/git/pmix/install/2.1
Is this possible ...
- 195
1
vote
0
answers
114
views
Is it possible to run users' jobs in chroot environment using Slurm
I'm using Slurm. I want users only to access their own /home/ directory (or directory that they executed their job), ubuntu packages that is installed by the sudo user, and binary files user /bin, ...
- 449
2
votes
2
answers
4k
views
"#" (comments) is SLURM job submissions
The question is related to the article Introducing SLURM by Princeton Research Computing.
For instance,
#SBATCH --job-name=slurm-test # create a short name for your job
SBATCH after the first # ...
- 123
1
vote
1
answer
155
views
Run a shell script within an already running shell script
I have a small SLURM cluster, which I am using to run jobs. I run a script and while that script is running, I want to enter a specific key command "t + enter". I believe I should do it by submitting ...
- 111
2
votes
1
answer
1k
views
Slurm on desktop computers, how to prioritize the owner
In short: On a Slurm cluster, I need some computers to be available and responsive to their respective owners during work hours.
Problem: I manage a small (but growing) heterogeneous cluster with ...
- 151
0
votes
0
answers
843
views
SLURM partition configuration: suspend and requeue
I am struggling a bit to configure our SLURM server. I wanted to have several queues (partitions) so that the short queues had higher priority and jobs could enter immediately suspending or requeuing ...
- 1
1
vote
2
answers
1k
views
Does the "watch" command put stress on a scheduler?
I have an account on a compute cluster which uses the SLURM scheduler. I have some jobs in the queue and I'm using the "watch" command to see their status:
watch squeue -u myUserName
Does constantly ...
0
votes
1
answer
4k
views
How to execute a script on several nodes using Slurm?
I have a R code that I want to execute on several nodes using Slurm, with each iteration of my paramater which goes on a node. This is my Slurm code :
#!/bin/bash
#SBATCH -o job-%A_task.out
#SBATCH -...
- 101
1
vote
0
answers
109
views
How to get CPU "load" of subset of CPU cores?
In my environment, each job is assigned (cpuset) to a subset of CPU cores, almost exclusively. (Actually, it's done by Slurm.)
Since the CPU core occupation is exclusive, can I get/calculate the 1-...
- 274