Questions tagged [slurm]
SLURM is a workload manager for Linux clusters
0
votes
0answers
12 views
SLURM partition configuration: suspend and requeue
I am struggling a bit to configure our SLURM server. I wanted to have several queues (partitions) so that the short queues had higher priority and jobs could enter immediately suspending or requeuing ...
0
votes
1answer
27 views
Does the “watch” command put stress on a scheduler?
I have an account on a compute cluster which uses the SLURM scheduler. I have some jobs in the queue and I'm using the "watch" command to see their status:
watch squeue -u myUserName
Does constantly ...
0
votes
1answer
20 views
How to execute a script on several nodes using Slurm?
I have a R code that I want to execute on several nodes using Slurm, with each iteration of my paramater which goes on a node. This is my Slurm code :
#!/bin/bash
#SBATCH -o job-%A_task.out
#SBATCH -...
0
votes
0answers
26 views
How to split a bash loop on several nodes with Slurm?
I would like to split this loop :
for i in $subset
do
srun Rscript my_script.R --subset $i
done
where $subset corresponds to seq 1 number . I would like to execute each value of i on multiple nodes....
0
votes
0answers
13 views
How to call a R variable in a loop with Slurm? [migrated]
I have a R (RHO_COR.R) script and I would like to create a loop in order to split the jobs on several nodes.
I show the part of the script where I would like to create the loop.
res <- foreach(i =...
1
vote
0answers
28 views
How to get CPU “load” of subset of CPU cores?
In my environment, each job is assigned (cpuset) to a subset of CPU cores, almost exclusively. (Actually, it's done by Slurm.)
Since the CPU core occupation is exclusive, can I get/calculate the 1-...
0
votes
0answers
19 views
Output name of parallel jobs on the same node using Slurm
I am using a small cluster with 4 nodes containing two GPUs each. I am quite new to Slurm so I hope I understood everything correctly.
When I submit a job with sbatch, a whole node gets reserved ...
0
votes
0answers
31 views
How to run script in a reserved node in slurm
I want to launch my script in Slurm's sbatch command that is already reserved a node(s). In OAR case once I reserved a node, I usually run my script (myscript.sh with some variable that is needed ...
1
vote
1answer
51 views
restore $0 or $BASH_SOURCE after it is modified by the cluster
I am using a shared SLURM cluster. I am trying to get the path of the bash script from inside the script itself.
There is already an excellent thread here: https://stackoverflow.com/questions/59895/...
0
votes
1answer
107 views
SLURM: How to determine maximum --cpus-per-task and --mem-per-cpu?
How can I determine the optimum/maximum number of CPUs per task when running a job? Is there a way to display the total available memory on a given CPU as well?
0
votes
0answers
36 views
Using `ipdb` to debug Python scripts run as Slurm jobs
I sometimes use import ipdb; ipdb.set_trace() in order to debug Python code with iPython (as described here: Step-by-step debugging with IPython). I also sometimes submit jobs to a computing cluster ...
0
votes
1answer
66 views
Maximum timeout for process
I have a script which I think can take few days to finish. Basically it submits slurm jobs when queue isn't full and in the process it can wait a while.
The script results in incomplete results, ...
2
votes
0answers
82 views
Running a slurm job array [closed]
I have a job_array.sh with 23 tasks:
> angsd -i ./realigned_RSFV1A.bam -anc ./BF_genome.fa -gl 1 -doSaf 1 -doSnpStat 1 -doMajorMinor 1 -doMaf 1 -doGeno 6 -doCounts 1 -doPost 1 -nThreads 24 -out ./...
1
vote
2answers
60 views
How to list all the processes assigned to a user on a cluster
We have a cluster in our university controlled by SLURM. The issue is that I have observed sometimes some of the resources are not being realised even though they do not show up in the squeue command. ...
3
votes
2answers
384 views
`watch` command with piping `|` [duplicate]
I want to keep monitoring a specific job on a slurm worload like cluster. I tried to use the watch command and grep the specific id. If the job id is4138, I tried
$> watch squeue -u mnyber004 | ...
2
votes
1answer
214 views
Watch-command-alias-expansion AND need to use quotes
My questions is similar to the watch question here but with a twist. I need to use quotes, which seem to be stripped by an aliased watch.
I want run watch on a custom slurm squeue command:
$alias ...
3
votes
1answer
5k views
How to submit a job to a specific node using Slurm's sbatch command?
Our nodes are named node001 ... node0xx in our cluster. I wonder, is it possible to submit a job to a specific node using Slurm's sbatch command? If so, can someone post an example code for that?
2
votes
0answers
106 views
SLURM: restrict GPU access only to SLURM
I have a single machine (Ubuntu 16.04 Server) with
4 TitanX GPUs. This will be a lab machine on which
students will learn about CUDA and stuff. I installed
SLURM because I want a tool to schedule and ...
0
votes
1answer
374 views
How to use a variable with SLURM sbatch to set the output/error file name?
We've just switched to using SLURM and I would like to submit a series of jobs using a loop and sbatch. Previously, I could use a variable as part of the output file names. I've been trying to do this ...
0
votes
1answer
809 views
How to cancel jobs on Slurm with job ID(job number) bigger than a certain number?
I have submitted 800 jobs on Slurm. I want to cancel those jobs that have job ID/number bigger than a number(since there is a mistake in them). I don't want to cancel all my jobs because some are ...
0
votes
1answer
283 views
launch singularity in a slurm script
I have singularity installed on /usr/local/bin and want to launch singularity from my slurm scirpt on my linux cluster but I keep getting this error "error:/var/spool/slurm/d/job01485/slurm_script: ...
1
vote
1answer
828 views
Best way to cancel all the SLURM jobs from shell command output
I submitted lots of SLURM job script with debug time limit (I forgot to change the time for actual run). Now they are all submitted at the same time, so they all start with job ID 197xxxxx. Now, I can ...
1
vote
0answers
320 views
How to increase memory limit and number of cores that Rstudio uses on a cluster
I launched Rstudio using FastX on Rivanna cluster (CentOS Linux release 7.3.1611 (Core)):
https://arcs.virginia.edu/rivanna
The main goal for doing that was to significantly increase the speed of ...
0
votes
1answer
446 views
slurm: srun & sbatch different performance with the same settings
In a slurm system, when I use srun command to run the program. It runs very slow and seems like only one processor works.
srun --pty -A free -J test -N 1 -n 1 -c 1 mpirun -np 16
$FEAPHOME8_3/...
1
vote
1answer
3k views
Automatically input “yes” on the bash file [closed]
Slurm is workload manager. There are two kinds of modes to run job, interactive(srun) and batch mode(sbatch).
When using interactive mode, one needs to leave the terminal open which may lead extra ...
2
votes
1answer
1k views
Check CPU/thread usage for a node in the Slurm job manager
I am working on a cluster machine that uses the Slurm job manager. I just started a multithreaded code and I would like to check the core and thread usage for a given node ID. For example,
scoreusage ...
3
votes
1answer
1k views
How to check SLURM environmental variables programmatically?
How can I programmatically access SLURM environmental variables, like MaxArraySize or MaxJobCount? I would like to partition my job arrays into chunks of the allowed maximum size. Can this information ...
3
votes
1answer
1k views
Possible effects of slurmstepd: error: Exceeded step memory limit at some point?
I have a question for those of you familiar with the scheduler Slurm. Sometimes I get the following error message slurmstepd: error: Exceeded step memory limit at some point.
I know it means the ...
1
vote
1answer
243 views
Submitting list of jobs to slurm
I have a large number of scripts that can be run as separate jobs on a computing cluster, which uses slurm. I want to select some of them, based on the contents, to submit. It's easy to identify the ...
4
votes
1answer
781 views
Running GNU Parallel on 2 or more nodes with Slurm scheduler
I am trying to distribute independent runs of a process using GNU Parallel on a HPC that uses Slurm workload manager. Briefly, here is the data analysis set up:
Script#1: myCommands
./myscript --...
6
votes
2answers
7k views
SLURM: Custom standard output name
When running a SLURM job using sbatch, slurm produces a standard output file which looks like slurm-102432.out (slurm-jobid.out). I would like to customise this to (yyyymmddhhmmss-jobid-jobname.txt). ...
1
vote
2answers
329 views
Text processing - Building a slurm topology.conf file from ibnetdiscover output
First things first: no knowledge of either slurm or Infiniband is required - this is a purely text processing problem.
Second - I'm aware of ib2slurm - the code is somehow broken and quite possibly ...
0
votes
0answers
135 views
How to restore sbatch array job output file names to their original names
In order to submit array jobs using Slurm, I've renamed hundreds of files with incremental numbers using the following command:
COUNT=1
for file in ./*; do
mv "$file" "${COUNT}"
let COUNT++
...
0
votes
1answer
89 views
How to see more information for queued Jobs in SLURM
With srun --test-only SLURM will return expected start date AND expected nodes.
However, if the job is submitted, I cannot get the information about the nodes. Is there any way to find out, what ...
3
votes
1answer
193 views
Using SLURM without a feature
Suppose my super computer has the following NODELIST's with the included features:
NODELIST FEATURES
NodeA (none)
NodeB specialfeature
and I am trying to benchmark performance ...
3
votes
1answer
1k views
Command Line Macros in Comments using SLURM
In my opinion, comments are comments are comments. They should NEVER change program state. Sadly, the people over at SLURM disagree.
SLURM requires the following syntax (or something similar) to ...
1
vote
0answers
611 views
xvfb-run aborting on cluster
I'm trying to run a python script with xvfb-run on a cluster using SLURM; when I use an interactive session (srun) it works fine, but when submitting to a queue (sbatch) it aborts/fails without much ...
