Questions tagged [slurm]
SLURM is a workload manager for Linux clusters
1
vote
0answers
24 views
How to get CPU “load” of subset of CPU cores?
In my environment, each job is assigned (cpuset) to a subset of CPU cores, almost exclusively. (Actually, it's done by Slurm.)
Since the CPU core occupation is exclusive, can I get/calculate the 1-...
0
votes
0answers
11 views
Output name of parallel jobs on the same node using Slurm
I am using a small cluster with 4 nodes containing two GPUs each. I am quite new to Slurm so I hope I understood everything correctly.
When I submit a job with sbatch, a whole node gets reserved ...
0
votes
0answers
26 views
How to run script in a reserved node in slurm
I want to launch my script in Slurm's sbatch command that is already reserved a node(s). In OAR case once I reserved a node, I usually run my script (myscript.sh with some variable that is needed ...
1
vote
1answer
45 views
restore $0 or $BASH_SOURCE after it is modified by the cluster
I am using a shared SLURM cluster. I am trying to get the path of the bash script from inside the script itself.
There is already an excellent thread here: https://stackoverflow.com/questions/59895/...
0
votes
1answer
47 views
SLURM: How to determine maximum --cpus-per-task and --mem-per-cpu?
How can I determine the optimum/maximum number of CPUs per task when running a job? Is there a way to display the total available memory on a given CPU as well?
0
votes
0answers
29 views
Using `ipdb` to debug Python scripts run as Slurm jobs
I sometimes use import ipdb; ipdb.set_trace() in order to debug Python code with iPython (as described here: Step-by-step debugging with IPython). I also sometimes submit jobs to a computing cluster ...
0
votes
1answer
47 views
Maximum timeout for process
I have a script which I think can take few days to finish. Basically it submits slurm jobs when queue isn't full and in the process it can wait a while.
The script results in incomplete results, ...
2
votes
0answers
50 views
Running a slurm job array [closed]
I have a job_array.sh with 23 tasks:
> angsd -i ./realigned_RSFV1A.bam -anc ./BF_genome.fa -gl 1 -doSaf 1 -doSnpStat 1 -doMajorMinor 1 -doMaf 1 -doGeno 6 -doCounts 1 -doPost 1 -nThreads 24 -out ./...
1
vote
2answers
52 views
How to list all the processes assigned to a user on a cluster
We have a cluster in our university controlled by SLURM. The issue is that I have observed sometimes some of the resources are not being realised even though they do not show up in the squeue command. ...
2
votes
2answers
342 views
`watch` command with piping `|` [duplicate]
I want to keep monitoring a specific job on a slurm worload like cluster. I tried to use the watch command and grep the specific id. If the job id is4138, I tried
$> watch squeue -u mnyber004 | ...
2
votes
1answer
170 views
Watch-command-alias-expansion AND need to use quotes
My questions is similar to the watch question here but with a twist. I need to use quotes, which seem to be stripped by an aliased watch.
I want run watch on a custom slurm squeue command:
$alias ...
2
votes
1answer
3k views
How to submit a job to a specific node using Slurm's sbatch command?
Our nodes are named node001 ... node0xx in our cluster. I wonder, is it possible to submit a job to a specific node using Slurm's sbatch command? If so, can someone post an example code for that?
2
votes
0answers
88 views
SLURM: restrict GPU access only to SLURM
I have a single machine (Ubuntu 16.04 Server) with
4 TitanX GPUs. This will be a lab machine on which
students will learn about CUDA and stuff. I installed
SLURM because I want a tool to schedule and ...
0
votes
1answer
307 views
How to use a variable with SLURM sbatch to set the output/error file name?
We've just switched to using SLURM and I would like to submit a series of jobs using a loop and sbatch. Previously, I could use a variable as part of the output file names. I've been trying to do this ...
0
votes
1answer
679 views
How to cancel jobs on Slurm with job ID(job number) bigger than a certain number?
I have submitted 800 jobs on Slurm. I want to cancel those jobs that have job ID/number bigger than a number(since there is a mistake in them). I don't want to cancel all my jobs because some are ...
0
votes
1answer
253 views
launch singularity in a slurm script
I have singularity installed on /usr/local/bin and want to launch singularity from my slurm scirpt on my linux cluster but I keep getting this error "error:/var/spool/slurm/d/job01485/slurm_script: ...
1
vote
1answer
506 views
Best way to cancel all the SLURM jobs from shell command output
I submitted lots of SLURM job script with debug time limit (I forgot to change the time for actual run). Now they are all submitted at the same time, so they all start with job ID 197xxxxx. Now, I can ...
1
vote
0answers
279 views
How to increase memory limit and number of cores that Rstudio uses on a cluster
I launched Rstudio using FastX on Rivanna cluster (CentOS Linux release 7.3.1611 (Core)):
https://arcs.virginia.edu/rivanna
The main goal for doing that was to significantly increase the speed of ...
0
votes
1answer
417 views
slurm: srun & sbatch different performance with the same settings
In a slurm system, when I use srun command to run the program. It runs very slow and seems like only one processor works.
srun --pty -A free -J test -N 1 -n 1 -c 1 mpirun -np 16
$FEAPHOME8_3/...
1
vote
1answer
3k views
Automatically input “yes” on the bash file [closed]
Slurm is workload manager. There are two kinds of modes to run job, interactive(srun) and batch mode(sbatch).
When using interactive mode, one needs to leave the terminal open which may lead extra ...
2
votes
1answer
968 views
Check CPU/thread usage for a node in the Slurm job manager
I am working on a cluster machine that uses the Slurm job manager. I just started a multithreaded code and I would like to check the core and thread usage for a given node ID. For example,
scoreusage ...
3
votes
1answer
996 views
How to check SLURM environmental variables programmatically?
How can I programmatically access SLURM environmental variables, like MaxArraySize or MaxJobCount? I would like to partition my job arrays into chunks of the allowed maximum size. Can this information ...
3
votes
1answer
1k views
Possible effects of slurmstepd: error: Exceeded step memory limit at some point?
I have a question for those of you familiar with the scheduler Slurm. Sometimes I get the following error message slurmstepd: error: Exceeded step memory limit at some point.
I know it means the ...
1
vote
1answer
230 views
Submitting list of jobs to slurm
I have a large number of scripts that can be run as separate jobs on a computing cluster, which uses slurm. I want to select some of them, based on the contents, to submit. It's easy to identify the ...
5
votes
1answer
720 views
Running GNU Parallel on 2 or more nodes with Slurm scheduler
I am trying to distribute independent runs of a process using GNU Parallel on a HPC that uses Slurm workload manager. Briefly, here is the data analysis set up:
Script#1: myCommands
./myscript --...
7
votes
2answers
6k views
SLURM: Custom standard output name
When running a SLURM job using sbatch, slurm produces a standard output file which looks like slurm-102432.out (slurm-jobid.out). I would like to customise this to (yyyymmddhhmmss-jobid-jobname.txt). ...
1
vote
2answers
324 views
Text processing - Building a slurm topology.conf file from ibnetdiscover output
First things first: no knowledge of either slurm or Infiniband is required - this is a purely text processing problem.
Second - I'm aware of ib2slurm - the code is somehow broken and quite possibly ...
0
votes
0answers
122 views
How to restore sbatch array job output file names to their original names
In order to submit array jobs using Slurm, I've renamed hundreds of files with incremental numbers using the following command:
COUNT=1
for file in ./*; do
mv "$file" "${COUNT}"
let COUNT++
...
0
votes
1answer
84 views
How to see more information for queued Jobs in SLURM
With srun --test-only SLURM will return expected start date AND expected nodes.
However, if the job is submitted, I cannot get the information about the nodes. Is there any way to find out, what ...
3
votes
1answer
190 views
Using SLURM without a feature
Suppose my super computer has the following NODELIST's with the included features:
NODELIST FEATURES
NodeA (none)
NodeB specialfeature
and I am trying to benchmark performance ...
3
votes
1answer
1k views
Command Line Macros in Comments using SLURM
In my opinion, comments are comments are comments. They should NEVER change program state. Sadly, the people over at SLURM disagree.
SLURM requires the following syntax (or something similar) to ...
1
vote
0answers
600 views
xvfb-run aborting on cluster
I'm trying to run a python script with xvfb-run on a cluster using SLURM; when I use an interactive session (srun) it works fine, but when submitting to a queue (sbatch) it aborts/fails without much ...
