Skip to main content

Questions tagged [slurm]

SLURM is a workload manager for Linux clusters

Filter by
Sorted by
Tagged with
0 votes
1 answer
15 views

SLURM_* variables causing syntax errors in multi-node HPC environment

My set up is as follows: A bash script that accepts a number of parameter (like how many nodes to use, which model to train, etc.). This is the job that I submit to the SLURM job queue This bash ...
Bram Vanroy's user avatar
0 votes
0 answers
23 views

SLURM job script - why is the tmp local directory deleted before archiving can occur? How to prevent this?

I wrote a SLURM job script to run a computational chemistry calculation using the CREST program (part of the xtb software package). In the script, I create a temporary directory on the local storage ...
lay lay's user avatar
0 votes
1 answer
26 views

slurm - minimizing effect of offline CPUs

I am doing experiments seeing how slurm behaves when it finds offline CPUs. In my experiments, slurm provides configurations that make available too few CPUs. Here's a few examples from an 8-cpu node ...
eftshift0's user avatar
  • 605
0 votes
1 answer
45 views

slurm - is it possible to query slurmctld/slurmd to know if they are using the right slurm.conf version?

I am facing a problem where slurmctld and slurmd are not in sync in terms of using the same slurm.conf file so we have this: error: Node node1 appears to have a different slurm.conf than the slurmctld....
eftshift0's user avatar
  • 605
1 vote
1 answer
33 views

qsub-like behavior for a slurm cluster

I recently switched to slurm and looking for a job submission tool, that behaves similar to qsub: It takes input through a pipe It prints the output to stdout Example: for n in `seq 1 10`; do ...
LazyCat's user avatar
  • 188
0 votes
2 answers
254 views

Unable to install Slurm on PC

I am trying to install slurm on Ubuntu PC. Therefore, I followed the instructions given over here I did the following - sudo apt update -y sudo apt install slurmd slurmctld -y mkdir sudo /etc/slurm-...
desert_ranger's user avatar
0 votes
0 answers
94 views

The for loop in bash runs infinitely when run using slurm

I have a bash script, called edit.sh that has this block of code: z=$(tail -1 mol-pos-1.xyz | awk '{print $NF}') echo "before loop: z is $z" >> log for ((r=0; $(echo "$z>20&...
Pauling0304's user avatar
0 votes
1 answer
196 views

How to reconnect to disconnected slurm compute node?

I started a compute node using srun on a SLURM cluster. My SSH connection got severed but my node is still active and consuming GPU resources. How do i reconnect to this node?
JobHunter69's user avatar
0 votes
0 answers
88 views

output of cat when file contains asterisks and quotes [duplicate]

I have a command stored in a text file command which looks like this: command_name par1 par2 path/with/backslashes -option_1 3 -option_2 0.05 --option_3=* -p 'P001=A' -p 'P002=B' ... The three dots ...
Michele Ancis's user avatar
0 votes
1 answer
484 views

how to cancel several jobs based on the job name

I am running several jobs on a cluster, however I want cancel multiple jobs based on its name instead of the job id. I read the slurm documentation and I see I can cancel them using scancel -n jobname ...
Xdrake's user avatar
  • 103
0 votes
1 answer
67 views

bsub not working with while loop

I am trying to run the following command but I get the error: syntax error near unexpected token `do' command: bsub -q XXX -P YYY -J ZZZ -R "rusage[mem=10000,scr=5000]" -R "span[hosts=...
kaur's user avatar
  • 23
0 votes
0 answers
580 views

Troubleshooting Slurm cgroups Plugin on Ubuntu 22.04

I'm facing a challenging issue with the Slurm cgroups plugin on a system running Ubuntu 22.04. We're relatively new to Slurm and started using it for better resource management in complex computing ...
Francisco Maria Calisto's user avatar
1 vote
3 answers
459 views

wget — download multiple files over multiple nodes on a cluster

Hi there I'm trying to download a large number of files at once; 279 to be precise. These are large BAM (~90GB) each. The cluster where I'm working has several nodes and fortunately I can allocate ...
Matteo's user avatar
  • 113
0 votes
0 answers
73 views

Create a -resume option in my own bash pipeline

I am doing a bash pipeline which submit different SLURM jobs having dependencies (as explained here or below). If one job crashes, it has to redo all jobs before to succeed for the job that failed ...
Alice Laigle's user avatar
0 votes
0 answers
64 views

Bash script - make sure there is always N processes launched on slurm

Task Out of a list of M jobs/commands to launch on remote machines with the srun command, make sure there's always N tasks running or requested (N<=M), until all M jobs have been launched. ...
Michele Ancis's user avatar
0 votes
0 answers
68 views

API for querying all cgroups applied to a process in Linux

I'm working on some software that needs to know its own maximum accessible memory. On Linux, this will depend on the cgroups. From what I have gathered, a process can read from /proc/self/cgroup to ...
Migwell's user avatar
  • 417
0 votes
1 answer
154 views

Looking for the Nvida devices in my WSL2 Ubuntu implementation for SLURM GRES

I am trying something odd. I have a slurm cluster configured that has 4 compute nodes. 2 of which are Windows 11 machines running WSL2 and I have it working. right now, I am trying to add GPU support ...
Keven Scharaswak's user avatar
0 votes
0 answers
61 views

Bash script ignores all terminal input following an underscore [duplicate]

Goal My server manages jobs using SLURM. Each job in the queue has a job ID, which can be either a single integer or two integers separated by an underscore, e.g., 123_4. You can use scontrol show job ...
Jamin's user avatar
  • 1
0 votes
1 answer
285 views

Naming scheme for compute nodes in a slurm cluster?

I am planning to build a new computer cluster based on Debian 11, Slurm, and a diverse range of hardware. The cluster will boot from a network location using PXE. It is expected that the cluster will ...
j4nd3r53n's user avatar
  • 717
0 votes
1 answer
161 views

Slurm is making my python script wait in running status

When I submit the following Slurm file with sbatch, the job has an immediately running status. However, the Python script runs after about 10 minutes. Also, I tried the same with srun command, and ...
ort23's user avatar
  • 1
1 vote
0 answers
283 views

How to access `/scratch` directory of nodes 2 to N in SLURM when node 1 doesn't have access to it?

I'm running a SLURM job with several nodes, where the nodes are arranged in the order of 1, 2, 3, 4, 5 ... N. However, I'm facing an issue where the node 1 doesn't have access to the /scratch ...
Daniel Wiczew's user avatar
2 votes
0 answers
361 views

How does SLURM authentication and encryption work?

I'm looking into the security of SLURM. None of the documentation is very clear about how it works. So far I have ascertained that authentication of users is done using MUNGE. As far as I can tell ...
Timmmm's user avatar
  • 606
1 vote
1 answer
3k views

Running multiple SLURM jobs on the same GPU

So, I am by no means a sysadmin but I need to use an existing SLURM installation to launch a sizable amount of jobs (around 5000). The cluster is composed of 1 node with 10 GPUs (with 8GB of memory ...
jacky la mouette's user avatar
1 vote
1 answer
477 views

slurm: one compute nodes not in sinfo?

I am setting up slurm 22.05.6, slowly building a cluster. So far I have set up one server, vogon, and a node, ceres; this seem to work fine - I can start jobs with srun. The server is on Debian 11, ...
j4nd3r53n's user avatar
  • 717
-2 votes
1 answer
382 views

How to display only the number of jobs running in HPC by specific user?

I want to Display only the number of jobs running in the HPC related to my username! I don't wanna see all the jobs as the command squeue does! Thanks!
Mahedi's user avatar
  • 1
5 votes
1 answer
842 views

Is munge required for a single node slurm setup?

I'm installing slurm on a single server to be used for scheduling purposes among a small group of people. There is not now, nor will there ever be, an intent to scale beyond this single node. Is munge ...
drjrm3's user avatar
  • 2,085
1 vote
1 answer
914 views

run slurm job on already logged on nodes

Is it possible to run a slurm job on already logged on nodes of a cluster? Suppose I am already logged on nodes: casade01 casade02 casade03 so that I don't need to wait in a queue. If it matters, I ...
georg's user avatar
  • 131
2 votes
1 answer
313 views

How to split the content of a folder into subfolders

I would like to know if there is an easy way to automatically (programmatically, within a bash script) to split the content of a folder into subfolders with the parent folder. I have a folder let's ...
Assa Yeroslaviz's user avatar
0 votes
1 answer
2k views

How to find which node name is invalid in Slurm error: "sbatch: error: Batch job submission failed: Invalid node name specified"

I'm getting this error "sbatch: error: Batch job submission failed: Invalid node name specified" for this bash script #!/bin/bash -l #SBATCH --gpus=1 #SBATCH -p overcap #SBATCH -A overcap #...
Gooby's user avatar
  • 59
1 vote
1 answer
1k views

Slurm - How to cancel jobs that contain a word in the job_name

How do you cancel jobs in Slurm that contain a word in job name? So for example let's say I do squeue in the terminal and get all my current jobs that are running. JOBID PARTITION NAME USER ST ...
Gooby's user avatar
  • 59
2 votes
1 answer
519 views

Cancel slurm job range in zsh

I am able to cancel slurm jobs by typing something like the following: $ scancel 66421802_[11-20] On bash, this works fine. However, on zsh I get the following error: $ scancel 66421802_[11-20] zsh: ...
Skeleton Bow's user avatar
1 vote
0 answers
237 views

How does salloc --exclusive behave on a crowded cluster?

In case of salloc --exclusive, the slurm man page is quite optimistic that it will find a node that is empty enough to grant exclusive access to the submitter. What happens, though, if this is not the ...
H. Weirauch's user avatar
-1 votes
1 answer
203 views

How to set bash script parameter to file minus the extension

so I have this bash file called test.sh #!/bin/bash -l out_file=`basename "$0"` #SBATCH --gpus=1 #SBATCH -p long #SBATCH -o err_${out_file::-3}.out and I want to output the error to ...
Gooby's user avatar
  • 59
0 votes
0 answers
30 views

Small TaskQueue shared on two computers

There are two computers with 12 physical cores each. Computer A should accept jobs and distribute them among A and B I want to setup Computers A and B such that A will accept jobs (via ssh) and ...
infinitezero's user avatar
1 vote
0 answers
219 views

Scheduling difference manual execution vs Slurm srun/sbatch

tl;dr A CPU intensive workload ignores the nice values of other processes only when started via Slurm sbatch/srun (not when manually started). I am running a systemd service which periodically gathers ...
ntippman's user avatar
1 vote
1 answer
139 views

Running jobs on HPC cluster using SLURM

Suppose I have two Python scripts: test1.py and test2.py How do I write a SLURM script to run these files on two different nodes simultaneously? Note that: test1.py and test2.py are independent of ...
user493562's user avatar
0 votes
0 answers
647 views

Bash process gets killed in HPC

I am new to bash scripting and I am using the below script to automate my job submission. This scripts waits for the previous job to finish and automatically submits a new job. while true do ...
Adupa Vasista's user avatar
3 votes
2 answers
6k views

Passing Argument to Comment in .sh script

I am a beginner in the use of .sh scripts so please excuse my ignorance. This is my problem: To submit my jobs to our cluster the corresponding submit file has to contain a "slurm header" ...
stollenm's user avatar
  • 133
0 votes
1 answer
1k views

list out all computer node connected to master node

I have HPC access, connected to other HPC via local IP or just simple hostname (i.e., compute1 or node3). I can log in to them by typing "ssh node3" or its IP like "ssh 10.2.6.20" ...
Ravi Saini's user avatar
1 vote
1 answer
48 views

Tools for batching N commands over L scripts (for N≫L)?

Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to M jobs either running or pending at any one time, of ...
kjo's user avatar
  • 15.6k
1 vote
1 answer
357 views

cant pipe SLURM `module list`command

I am working on a SLURM cluster and there is a command to list all loaded software modules. I want to process the output and i.e. grep it for a certain word. However if I try to use the pipe I get ...
v.tralala's user avatar
  • 195
1 vote
1 answer
532 views

simulating user input in a bash script on slurm

I believe my problem is similar as found here but I was confused by the reply (and I can't comment in this question yet due to my low rep). I have a bash script I need to run using slurm, so when the ...
Diogo de Moraes's user avatar
2 votes
0 answers
234 views

Slurm multifactor priority - how to configure Slurm to consider past usage?

I'm trying to configure our Slurm to allocate users' jobs according to past usage. The goal is to have the job priority consider past time usage of mainly GPUs. I understand I must use the FairShare ...
itay1itzhak's user avatar
0 votes
0 answers
86 views

Slurm 18 allocates MPI jobs to the same CPUs on the node

For some reason when launching an mpi job with SLURM on CentOS8 cluster, slurm ties mpi processes to CPUs always starting from CPU0. Say there are 128 CPU cores on a compute node. I launch mpi job ...
Alex P's user avatar
  • 81
2 votes
1 answer
914 views

Python error only when I run script on Linux cluster: _tkinter.TclError: no display name and no $DISPLAY environment variable

My question is related to a python error, but I suspect that it is more a Linux question than a python one. Thus I post it first here. I am running a python script which does a calculation and then ...
Britzel's user avatar
  • 165
0 votes
1 answer
2k views

How do I print slurm variables to standard slurm output?

I want to print the following variables to the standard output file slurm-XXXXX.out produced by Slurm. Right now, I am generating a separate .info file for every job. echo "SLURM Job ID : $...
nac001's user avatar
  • 103
2 votes
1 answer
243 views

Very new to slurm. How to get slurm to run multiple core jobs on my linux cluster?

I've been trying to move some existing processes to a revamped linux cluster that now runs on slurm. I thought I have it done, but my problem now is trying to get multiple cores to run. Here is my ...
ted y's user avatar
  • 21
1 vote
1 answer
672 views

Why it's not possible to run wget with background option in slurm script?

I used this script for downloading files. Without -b, wget download files one by one. With -b, I have the possibility to download files in background but also simultaneously. Unfortunately, the script ...
S.AZOUZ's user avatar
  • 11
1 vote
1 answer
602 views

How to use Active Directory (domain groups) for user management in SLURM?

We have a GPU cluster (centOS 8) in our company network which runs SLURM. Now we would like to realize the SLURM user management via our windows domain controller (active directory). Every user which ...
Spenhouet's user avatar
  • 111
0 votes
1 answer
423 views

SLURM: How do I submit multilple OpenMP parallel codes in a job script

I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.) #!/bin/bash #SBATCH -N 1 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=2 export OMP_NUM_THREADS=...
symanzik138's user avatar