Questions tagged [slurm]

SLURM is a workload manager for Linux clusters

Filter by
Sorted by
Tagged with
0 votes
0 answers
81 views

Contradiction in GPU numbering of `$CUDA_VISIBLE_DEVICES` and `nvidia-smi` when MIG is active

My aim is very simple. We have multiple GPUs on each node. However, if I allocate only two GPUs for me. nvidia-smi or nvidia-smi -L shows a list of all GPUs including those being used by others and ...
user avatar
0 votes
0 answers
10 views

Ways to supply common arguments to qsub/slurm using single file?

My team is building a small HPC pipeline that requires multiple cluster jobs to be submitted. So far, we've been using 5+ submission scripts, but this makes changing configuration options a hassle. We ...
user avatar
  • 13
0 votes
0 answers
25 views

Small TaskQueue shared on two computers

There are two computers with 12 physical cores each. Computer A should accept jobs and distribute them among A and B I want to setup Computers A and B such that A will accept jobs (via ssh) and ...
user avatar
0 votes
0 answers
13 views

Fixing the threads to cores in slurm (and avoid the OS moves them)

I am having problems with with the (infinity) of srun options. Does any one have a recipe to allocate threads to cores, and more importantly, avoid that there is migration of the threads among the ...
user avatar
  • 1
0 votes
0 answers
59 views

Scheduling difference manual execution vs Slurm srun/sbatch

tl;dr A CPU intensive workload ignores the nice values of other processes only when started via Slurm sbatch/srun (not when manually started). I am running a systemd service which periodically gathers ...
user avatar
0 votes
0 answers
14 views

Controlling HPC Slurm job dependencies by list of filenames

I am quite new to slurm and I am trying to set up a job dependencies pipeline using a master script called dependencies.sh, which contains something like: jid1=$(sbatch step1.sh) jid2=$(sbatch --...
user avatar
  • 101
1 vote
1 answer
30 views

Running jobs on HPC cluster using SLURM

Suppose I have two Python scripts: test1.py and test2.py How do I write a SLURM script to run these files on two different nodes simultaneously? Note that: test1.py and test2.py are independent of ...
user avatar
0 votes
0 answers
42 views

Bash process gets killed in HPC

I am new to bash scripting and I am using the below script to automate my job submission. This scripts waits for the previous job to finish and automatically submits a new job. while true do ...
user avatar
3 votes
2 answers
252 views

Passing Argument to Comment in .sh script

I am a beginner in the use of .sh scripts so please excuse my ignorance. This is my problem: To submit my jobs to our cluster the corresponding submit file has to contain a "slurm header" ...
user avatar
  • 133
0 votes
1 answer
82 views

list out all computer node connected to master node

I have HPC access, connected to other HPC via local IP or just simple hostname (i.e., compute1 or node3). I can log in to them by typing "ssh node3" or its IP like "ssh 10.2.6.20" ...
user avatar
1 vote
1 answer
40 views

Tools for batching N commands over L scripts (for N≫L)?

Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to M jobs either running or pending at any one time, of ...
user avatar
  • 13k
0 votes
1 answer
58 views

cant pipe SLURM `module list`command

I am working on a SLURM cluster and there is a command to list all loaded software modules. I want to process the output and i.e. grep it for a certain word. However if I try to use the pipe I get ...
user avatar
  • 165
1 vote
1 answer
91 views

simulating user input in a bash script on slurm

I believe my problem is similar as found here but I was confused by the reply (and I can't comment in this question yet due to my low rep). I have a bash script I need to run using slurm, so when the ...
user avatar
0 votes
0 answers
16 views

slurm: CPUs=8 versus Sockets=1 CoresPerSocket=4 ThreadsPerCore=2

i switched to the slurm batch system recently. i have a laptop with 1 socket, 4 cores, and 2 threads per core. when i set up slurm using explicity socket, core and thread info, i canNOT run 8 ...
user avatar
0 votes
0 answers
73 views

SLURM: "unsafe" code, forking and restarting processes which exceed memory limit

I am running code in the open-source computer algebra system Sage in a SLURM-managed multi-node cluster environment. The code is highly parallel: I use the Python package mpi4py to manage a task queue ...
user avatar
1 vote
0 answers
57 views

Slurm multifactor priority - how to configure Slurm to consider past usage?

I'm trying to configure our Slurm to allocate users' jobs according to past usage. The goal is to have the job priority consider past time usage of mainly GPUs. I understand I must use the FairShare ...
user avatar
0 votes
0 answers
24 views

Slurm 18 allocates MPI jobs to the same CPUs on the node

For some reason when launching an mpi job with SLURM on CentOS8 cluster, slurm ties mpi processes to CPUs always starting from CPU0. Say there are 128 CPU cores on a compute node. I launch mpi job ...
user avatar
  • 81
1 vote
1 answer
347 views

Python error only when I run script on Linux cluster: _tkinter.TclError: no display name and no $DISPLAY environment variable

My question is related to a python error, but I suspect that it is more a Linux question than a python one. Thus I post it first here. I am running a python script which does a calculation and then ...
user avatar
  • 155
0 votes
1 answer
310 views

How do I print slurm variables to standard slurm output?

I want to print the following variables to the standard output file slurm-XXXXX.out produced by Slurm. Right now, I am generating a separate .info file for every job. echo "SLURM Job ID : $...
user avatar
  • 103
1 vote
1 answer
33 views

Very new to slurm. How to get slurm to run multiple core jobs on my linux cluster?

I've been trying to move some existing processes to a revamped linux cluster that now runs on slurm. I thought I have it done, but my problem now is trying to get multiple cores to run. Here is my ...
user avatar
  • 11
1 vote
1 answer
364 views

Why it's not possible to run wget with background option in slurm script?

I used this script for downloading files. Without -b, wget download files one by one. With -b, I have the possibility to download files in background but also simultaneously. Unfortunately, the script ...
user avatar
  • 11
1 vote
1 answer
175 views

How to use Active Directory (domain groups) for user management in SLURM?

We have a GPU cluster (centOS 8) in our company network which runs SLURM. Now we would like to realize the SLURM user management via our windows domain controller (active directory). Every user which ...
user avatar
  • 111
0 votes
1 answer
96 views

SLURM: How do I submit multilple OpenMP parallel codes in a job script

I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.) #!/bin/bash #SBATCH -N 1 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=2 export OMP_NUM_THREADS=...
user avatar
0 votes
1 answer
302 views

How to save the output of slurm command into a file?

I have installed slurm command to track the amount of data consumed by an interface apt install slurm My gold is to save its output into a file, I wrote this script to let me do that, but I did not ...
user avatar
0 votes
1 answer
108 views

Canceling SLURM jobs without receiving email notifications

Given some SLURM jobs which were started using sbatch --mail-type=ALL, is there any way to cancel them without receiving the email notifications informing you that they were canceled? I was thinking ...
user avatar
  • 282
1 vote
2 answers
2k views

Passing configure option to rpmbuild?

To build a piece of software I normally do rpmbuild -ta slurm*.tar.bz2 However I now need to configure the software with the option --with-pmix=/home/user/git/pmix/install/2.1 Is this possible ...
user avatar
  • 123
1 vote
0 answers
73 views

Is it possible to run users' jobs in chroot environment using Slurm

I'm using Slurm. I want users only to access their own /home/ directory (or directory that they executed their job), ubuntu packages that is installed by the sudo user, and binary files user /bin, ...
user avatar
  • 229
2 votes
2 answers
2k views

"#" (comments) is SLURM job submissions

The question is related to the article Introducing SLURM by Princeton Research Computing. For instance, #SBATCH --job-name=slurm-test # create a short name for your job SBATCH after the first # ...
user avatar
  • 123
1 vote
1 answer
58 views

Run a shell script within an already running shell script

I have a small SLURM cluster, which I am using to run jobs. I run a script and while that script is running, I want to enter a specific key command "t + enter". I believe I should do it by submitting ...
user avatar
  • 111
2 votes
1 answer
731 views

Slurm on desktop computers, how to prioritize the owner

In short: On a Slurm cluster, I need some computers to be available and responsive to their respective owners during work hours. Problem: I manage a small (but growing) heterogeneous cluster with ...
user avatar
  • 141
0 votes
0 answers
477 views

SLURM partition configuration: suspend and requeue

I am struggling a bit to configure our SLURM server. I wanted to have several queues (partitions) so that the short queues had higher priority and jobs could enter immediately suspending or requeuing ...
user avatar
  • 1
0 votes
2 answers
720 views

Does the "watch" command put stress on a scheduler?

I have an account on a compute cluster which uses the SLURM scheduler. I have some jobs in the queue and I'm using the "watch" command to see their status: watch squeue -u myUserName Does constantly ...
user avatar
0 votes
1 answer
2k views

How to execute a script on several nodes using Slurm?

I have a R code that I want to execute on several nodes using Slurm, with each iteration of my paramater which goes on a node. This is my Slurm code : #!/bin/bash #SBATCH -o job-%A_task.out #SBATCH -...
user avatar
  • 101
1 vote
0 answers
81 views

How to get CPU "load" of subset of CPU cores?

In my environment, each job is assigned (cpuset) to a subset of CPU cores, almost exclusively. (Actually, it's done by Slurm.) Since the CPU core occupation is exclusive, can I get/calculate the 1-...
user avatar
  • 274
3 votes
1 answer
243 views

restore $0 or $BASH_SOURCE after it is modified by the cluster

I am using a shared SLURM cluster. I am trying to get the path of the bash script from inside the script itself. There is already an excellent thread here: https://stackoverflow.com/questions/59895/...
user avatar
  • 199
1 vote
1 answer
764 views

SLURM: How to determine maximum --cpus-per-task and --mem-per-cpu?

How can I determine the optimum/maximum number of CPUs per task when running a job? Is there a way to display the total available memory on a given CPU as well?
user avatar
0 votes
1 answer
690 views

Maximum timeout for process

I have a script which I think can take few days to finish. Basically it submits slurm jobs when queue isn't full and in the process it can wait a while. The script results in incomplete results, ...
user avatar
  • 101
2 votes
0 answers
456 views

Running a slurm job array [closed]

I have a job_array.sh with 23 tasks: > angsd -i ./realigned_RSFV1A.bam -anc ./BF_genome.fa -gl 1 -doSaf 1 -doSnpStat 1 -doMajorMinor 1 -doMaf 1 -doGeno 6 -doCounts 1 -doPost 1 -nThreads 24 -out ./...
user avatar
1 vote
2 answers
1k views

How to list all the processes assigned to a user on a cluster

We have a cluster in our university controlled by SLURM. The issue is that I have observed sometimes some of the resources are not being realised even though they do not show up in the squeue command. ...
user avatar
  • 379
4 votes
2 answers
1k views

`watch` command with piping `|` [duplicate]

I want to keep monitoring a specific job on a slurm worload like cluster. I tried to use the watch command and grep the specific id. If the job id is4138, I tried $> watch squeue -u mnyber004 | ...
user avatar
  • 257
3 votes
1 answer
1k views

Watch-command-alias-expansion AND need to use quotes

My questions is similar to the watch question here but with a twist. I need to use quotes, which seem to be stripped by an aliased watch. I want run watch on a custom slurm squeue command: $alias ...
user avatar
21 votes
1 answer
36k views

How to submit a job to a specific node using Slurm's sbatch command?

Our nodes are named node001 ... node0xx in our cluster. I wonder, is it possible to submit a job to a specific node using Slurm's sbatch command? If so, can someone post an example code for that?
user avatar
  • 659
3 votes
0 answers
291 views

SLURM: restrict GPU access only to SLURM

I have a single machine (Ubuntu 16.04 Server) with 4 TitanX GPUs. This will be a lab machine on which students will learn about CUDA and stuff. I installed SLURM because I want a tool to schedule and ...
user avatar
  • 1,171
0 votes
1 answer
2k views

How to use a variable with SLURM sbatch to set the output/error file name?

We've just switched to using SLURM and I would like to submit a series of jobs using a loop and sbatch. Previously, I could use a variable as part of the output file names. I've been trying to do this ...
user avatar
  • 1
12 votes
2 answers
12k views

How to cancel jobs on Slurm with job ID(job number) bigger than a certain number?

I have submitted 800 jobs on Slurm. I want to cancel those jobs that have job ID/number bigger than a number(since there is a mistake in them). I don't want to cancel all my jobs because some are ...
user avatar
0 votes
1 answer
663 views

launch singularity in a slurm script

I have singularity installed on /usr/local/bin and want to launch singularity from my slurm scirpt on my linux cluster but I keep getting this error "error:/var/spool/slurm/d/job01485/slurm_script: ...
user avatar
23 votes
5 answers
24k views

Best way to cancel all the SLURM jobs from shell command output

I submitted lots of SLURM job script with debug time limit (I forgot to change the time for actual run). Now they are all submitted at the same time, so they all start with job ID 197xxxxx. Now, I can ...
user avatar
1 vote
0 answers
467 views

How to increase memory limit and number of cores that Rstudio uses on a cluster

I launched Rstudio using FastX on Rivanna cluster (CentOS Linux release 7.3.1611 (Core)): https://arcs.virginia.edu/rivanna The main goal for doing that was to significantly increase the speed of ...
user avatar
0 votes
1 answer
1k views

slurm: srun & sbatch different performance with the same settings

In a slurm system, when I use srun command to run the program. It runs very slow and seems like only one processor works. srun --pty -A free -J test -N 1 -n 1 -c 1 mpirun -np 16 $FEAPHOME8_3/...
user avatar
2 votes
1 answer
11k views

Automatically input "yes" on the bash file [closed]

Slurm is workload manager. There are two kinds of modes to run job, interactive(srun) and batch mode(sbatch). When using interactive mode, one needs to leave the terminal open which may lead extra ...
user avatar