Questions tagged [slurm]

SLURM is a workload manager for Linux clusters

Filter by
Sorted by
Tagged with
0 votes
0 answers
27 views

Looking for the Nvida devices in my WSL2 Ubuntu implementation for SLURM GRES

I am trying something odd. I have a slurm cluster configured that has 4 compute nodes. 2 of which are Windows 11 machines running WSL2 and I have it working. right now, I am trying to add GPU support ...
Keven Scharaswak's user avatar
0 votes
0 answers
31 views

Bash script ignores all terminal input following an underscore [duplicate]

Goal My server manages jobs using SLURM. Each job in the queue has a job ID, which can be either a single integer or two integers separated by an underscore, e.g., 123_4. You can use scontrol show job ...
Jamin's user avatar
  • 1
0 votes
0 answers
20 views

Slurm not preserving QoS limits

I am trying to set QoS limits on the cluster, to be specific, I want to make sure nobody is able to submit jobs to a specific partition. So I enable CPU limits on the partition and it works as ...
Nobody's user avatar
  • 93
0 votes
0 answers
16 views

Changing SSH keys on the compute node in pcluster using slurm

Our login-node is currently one of the compute node, residing in a separate queue. We want to setup custom SSH keys on /etc/ssh for that login-node. We tried pulling in the custom SSH keys on the post-...
Nobody's user avatar
  • 93
0 votes
1 answer
49 views

Naming scheme for compute nodes in a slurm cluster?

I am planning to build a new computer cluster based on Debian 11, Slurm, and a diverse range of hardware. The cluster will boot from a network location using PXE. It is expected that the cluster will ...
j4nd3r53n's user avatar
  • 591
0 votes
1 answer
57 views

Slurm is making my python script wait in running status

When I submit the following Slurm file with sbatch, the job has an immediately running status. However, the Python script runs after about 10 minutes. Also, I tried the same with srun command, and ...
ort23's user avatar
  • 1
1 vote
0 answers
75 views

How to access `/scratch` directory of nodes 2 to N in SLURM when node 1 doesn't have access to it?

I'm running a SLURM job with several nodes, where the nodes are arranged in the order of 1, 2, 3, 4, 5 ... N. However, I'm facing an issue where the node 1 doesn't have access to the /scratch ...
Daniel Wiczew's user avatar
0 votes
0 answers
44 views

Use more CPU cores using code in PyTorch that also uses GPU

I'm trying to successfully run code in PyTorch that uses DataLoader. It is possible to configure the DataLoader to load data using several processes (which speeds up data loading a lot), via the use ...
Marco's user avatar
  • 1
0 votes
0 answers
319 views

Using a node by its IP address

I was trying to use a specific node by its IP address during a slurm job. I know we can specify the node inside the .sh file by using the #SBATCH --nodelist =ncbxxx command where we specify the names ...
Joy Acharjee's user avatar
2 votes
0 answers
145 views

How does SLURM authentication and encryption work?

I'm looking into the security of SLURM. None of the documentation is very clear about how it works. So far I have ascertained that authentication of users is done using MUNGE. As far as I can tell ...
Timmmm's user avatar
  • 537
1 vote
1 answer
792 views

Running multiple SLURM jobs on the same GPU

So, I am by no means a sysadmin but I need to use an existing SLURM installation to launch a sizable amount of jobs (around 5000). The cluster is composed of 1 node with 10 GPUs (with 8GB of memory ...
jacky la mouette's user avatar
1 vote
1 answer
268 views

slurm: one compute nodes not in sinfo?

I am setting up slurm 22.05.6, slowly building a cluster. So far I have set up one server, vogon, and a node, ceres; this seem to work fine - I can start jobs with srun. The server is on Debian 11, ...
j4nd3r53n's user avatar
  • 591
-2 votes
1 answer
193 views

How to display only the number of jobs running in HPC by specific user?

I want to Display only the number of jobs running in the HPC related to my username! I don't wanna see all the jobs as the command squeue does! Thanks!
Mahedi's user avatar
  • 1
0 votes
0 answers
33 views

Launch Program in Background from a Epilog Script

I want to start a program in the background from a epilog script. I put the following content into the epilog script: #! /bin/bash /usr/bin/nohup /path/to/my/program & I know the script is being ...
das_jasper's user avatar
5 votes
1 answer
416 views

Is munge required for a single node slurm setup?

I'm installing slurm on a single server to be used for scheduling purposes among a small group of people. There is not now, nor will there ever be, an intent to scale beyond this single node. Is munge ...
drjrm3's user avatar
  • 1,905
1 vote
1 answer
343 views

run slurm job on already logged on nodes

Is it possible to run a slurm job on already logged on nodes of a cluster? Suppose I am already logged on nodes: casade01 casade02 casade03 so that I don't need to wait in a queue. If it matters, I ...
georg's user avatar
  • 131
2 votes
1 answer
148 views

How to split the content of a folder into subfolders

I would like to know if there is an easy way to automatically (programmatically, within a bash script) to split the content of a folder into subfolders with the parent folder. I have a folder let's ...
Assa Yeroslaviz's user avatar
0 votes
1 answer
1k views

How to find which node name is invalid in Slurm error: "sbatch: error: Batch job submission failed: Invalid node name specified"

I'm getting this error "sbatch: error: Batch job submission failed: Invalid node name specified" for this bash script #!/bin/bash -l #SBATCH --gpus=1 #SBATCH -p overcap #SBATCH -A overcap #...
Gooby's user avatar
  • 59
1 vote
1 answer
407 views

Slurm - How to cancel jobs that contain a word in the job_name

How do you cancel jobs in Slurm that contain a word in job name? So for example let's say I do squeue in the terminal and get all my current jobs that are running. JOBID PARTITION NAME USER ST ...
Gooby's user avatar
  • 59
2 votes
1 answer
333 views

Cancel slurm job range in zsh

I am able to cancel slurm jobs by typing something like the following: $ scancel 66421802_[11-20] On bash, this works fine. However, on zsh I get the following error: $ scancel 66421802_[11-20] zsh: ...
Skeleton Bow's user avatar
1 vote
0 answers
134 views

How does salloc --exclusive behave on a crowded cluster?

In case of salloc --exclusive, the slurm man page is quite optimistic that it will find a node that is empty enough to grant exclusive access to the submitter. What happens, though, if this is not the ...
H. Weirauch's user avatar
-1 votes
1 answer
157 views

How to set bash script parameter to file minus the extension

so I have this bash file called test.sh #!/bin/bash -l out_file=`basename "$0"` #SBATCH --gpus=1 #SBATCH -p long #SBATCH -o err_${out_file::-3}.out and I want to output the error to ...
Gooby's user avatar
  • 59
0 votes
0 answers
30 views

Small TaskQueue shared on two computers

There are two computers with 12 physical cores each. Computer A should accept jobs and distribute them among A and B I want to setup Computers A and B such that A will accept jobs (via ssh) and ...
infinitezero's user avatar
1 vote
0 answers
147 views

Scheduling difference manual execution vs Slurm srun/sbatch

tl;dr A CPU intensive workload ignores the nice values of other processes only when started via Slurm sbatch/srun (not when manually started). I am running a systemd service which periodically gathers ...
ntippman's user avatar
1 vote
1 answer
114 views

Running jobs on HPC cluster using SLURM

Suppose I have two Python scripts: test1.py and test2.py How do I write a SLURM script to run these files on two different nodes simultaneously? Note that: test1.py and test2.py are independent of ...
user493562's user avatar
0 votes
0 answers
401 views

Bash process gets killed in HPC

I am new to bash scripting and I am using the below script to automate my job submission. This scripts waits for the previous job to finish and automatically submits a new job. while true do ...
Adupa Vasista's user avatar
3 votes
2 answers
3k views

Passing Argument to Comment in .sh script

I am a beginner in the use of .sh scripts so please excuse my ignorance. This is my problem: To submit my jobs to our cluster the corresponding submit file has to contain a "slurm header" ...
stollenm's user avatar
  • 133
0 votes
1 answer
693 views

list out all computer node connected to master node

I have HPC access, connected to other HPC via local IP or just simple hostname (i.e., compute1 or node3). I can log in to them by typing "ssh node3" or its IP like "ssh 10.2.6.20" ...
Ravi Saini's user avatar
1 vote
1 answer
44 views

Tools for batching N commands over L scripts (for N≫L)?

Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to M jobs either running or pending at any one time, of ...
kjo's user avatar
  • 14.8k
1 vote
1 answer
237 views

cant pipe SLURM `module list`command

I am working on a SLURM cluster and there is a command to list all loaded software modules. I want to process the output and i.e. grep it for a certain word. However if I try to use the pipe I get ...
v.tralala's user avatar
  • 185
1 vote
1 answer
344 views

simulating user input in a bash script on slurm

I believe my problem is similar as found here but I was confused by the reply (and I can't comment in this question yet due to my low rep). I have a bash script I need to run using slurm, so when the ...
Diogo de Moraes's user avatar
2 votes
0 answers
152 views

Slurm multifactor priority - how to configure Slurm to consider past usage?

I'm trying to configure our Slurm to allocate users' jobs according to past usage. The goal is to have the job priority consider past time usage of mainly GPUs. I understand I must use the FairShare ...
itay1itzhak's user avatar
0 votes
0 answers
60 views

Slurm 18 allocates MPI jobs to the same CPUs on the node

For some reason when launching an mpi job with SLURM on CentOS8 cluster, slurm ties mpi processes to CPUs always starting from CPU0. Say there are 128 CPU cores on a compute node. I launch mpi job ...
Alex P's user avatar
  • 71
2 votes
1 answer
656 views

Python error only when I run script on Linux cluster: _tkinter.TclError: no display name and no $DISPLAY environment variable

My question is related to a python error, but I suspect that it is more a Linux question than a python one. Thus I post it first here. I am running a python script which does a calculation and then ...
Britzel's user avatar
  • 165
0 votes
1 answer
1k views

How do I print slurm variables to standard slurm output?

I want to print the following variables to the standard output file slurm-XXXXX.out produced by Slurm. Right now, I am generating a separate .info file for every job. echo "SLURM Job ID : $...
nac001's user avatar
  • 103
2 votes
1 answer
150 views

Very new to slurm. How to get slurm to run multiple core jobs on my linux cluster?

I've been trying to move some existing processes to a revamped linux cluster that now runs on slurm. I thought I have it done, but my problem now is trying to get multiple cores to run. Here is my ...
ted y's user avatar
  • 21
1 vote
1 answer
569 views

Why it's not possible to run wget with background option in slurm script?

I used this script for downloading files. Without -b, wget download files one by one. With -b, I have the possibility to download files in background but also simultaneously. Unfortunately, the script ...
S.AZOUZ's user avatar
  • 11
1 vote
1 answer
421 views

How to use Active Directory (domain groups) for user management in SLURM?

We have a GPU cluster (centOS 8) in our company network which runs SLURM. Now we would like to realize the SLURM user management via our windows domain controller (active directory). Every user which ...
Spenhouet's user avatar
  • 111
0 votes
1 answer
291 views

SLURM: How do I submit multilple OpenMP parallel codes in a job script

I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.) #!/bin/bash #SBATCH -N 1 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=2 export OMP_NUM_THREADS=...
symanzik138's user avatar
0 votes
1 answer
1k views

How to save the output of slurm command into a file?

I have installed slurm command to track the amount of data consumed by an interface apt install slurm My gold is to save its output into a file, I wrote this script to let me do that, but I did not ...
Youssef's user avatar
0 votes
1 answer
264 views

Canceling SLURM jobs without receiving email notifications

Given some SLURM jobs which were started using sbatch --mail-type=ALL, is there any way to cancel them without receiving the email notifications informing you that they were canceled? I was thinking ...
Socob's user avatar
  • 331
1 vote
2 answers
3k views

Passing configure option to rpmbuild?

To build a piece of software I normally do rpmbuild -ta slurm*.tar.bz2 However I now need to configure the software with the option --with-pmix=/home/user/git/pmix/install/2.1 Is this possible ...
loris's user avatar
  • 195
1 vote
0 answers
114 views

Is it possible to run users' jobs in chroot environment using Slurm

I'm using Slurm. I want users only to access their own /home/ directory (or directory that they executed their job), ubuntu packages that is installed by the sudo user, and binary files user /bin, ...
alper's user avatar
  • 449
2 votes
2 answers
4k views

"#" (comments) is SLURM job submissions

The question is related to the article Introducing SLURM by Princeton Research Computing. For instance, #SBATCH --job-name=slurm-test # create a short name for your job SBATCH after the first # ...
AlphaF20's user avatar
  • 123
1 vote
1 answer
155 views

Run a shell script within an already running shell script

I have a small SLURM cluster, which I am using to run jobs. I run a script and while that script is running, I want to enter a specific key command "t + enter". I believe I should do it by submitting ...
George's user avatar
  • 111
2 votes
1 answer
1k views

Slurm on desktop computers, how to prioritize the owner

In short: On a Slurm cluster, I need some computers to be available and responsive to their respective owners during work hours. Problem: I manage a small (but growing) heterogeneous cluster with ...
DavidAce's user avatar
  • 151
0 votes
0 answers
843 views

SLURM partition configuration: suspend and requeue

I am struggling a bit to configure our SLURM server. I wanted to have several queues (partitions) so that the short queues had higher priority and jobs could enter immediately suspending or requeuing ...
GoNiS's user avatar
  • 1
1 vote
2 answers
1k views

Does the "watch" command put stress on a scheduler?

I have an account on a compute cluster which uses the SLURM scheduler. I have some jobs in the queue and I'm using the "watch" command to see their status: watch squeue -u myUserName Does constantly ...
Niagara Falls's user avatar
0 votes
1 answer
4k views

How to execute a script on several nodes using Slurm?

I have a R code that I want to execute on several nodes using Slurm, with each iteration of my paramater which goes on a node. This is my Slurm code : #!/bin/bash #SBATCH -o job-%A_task.out #SBATCH -...
Paillou's user avatar
  • 101
1 vote
0 answers
109 views

How to get CPU "load" of subset of CPU cores?

In my environment, each job is assigned (cpuset) to a subset of CPU cores, almost exclusively. (Actually, it's done by Slurm.) Since the CPU core occupation is exclusive, can I get/calculate the 1-...
noname's user avatar
  • 274