Questions tagged [batch-jobs]

Grid engine batch job processing (qsub, qstat, …)

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Use of makefile varriable in Batch script [closed]

I have a variable in my makefile (mybuild.mak). The makefile has a variable. i.e BUILD_PATH = D:/<some path> The variable BUILD_PATH is to be used in a batch file (RunTest.bat). How to use the ...
user avatar
0 votes
2 answers
30 views

ZSh script to batch process filenames containing spaces and substitute extension

Is there a zsh script that can convert a directory files in one music format and be converted into Ogg opus format. The filenames have spaces in their names. For example, a directory contains 10 ...
user avatar
  • 101
0 votes
1 answer
89 views

zsh: bad subscript for direct array assignment: 0

I have the following bashscript below. My goal is to itterate over multiple files in the directory. The name of the files will be batch_1, batch_2, batch_3, batch_4, etc. There should be no more than ...
user avatar
  • 1
0 votes
0 answers
14 views

Controlling HPC Slurm job dependencies by list of filenames

I am quite new to slurm and I am trying to set up a job dependencies pipeline using a master script called dependencies.sh, which contains something like: jid1=$(sbatch step1.sh) jid2=$(sbatch --...
user avatar
  • 101
0 votes
0 answers
19 views

In middle of the bash script next job/code takes time to trigger if first is completed

I have a script which have my codes: # scheduling details for abc.sas, first status check after 1 hr , status check interval is 30 mins execute "claimant.sas" echo "execute function ...
user avatar
6 votes
4 answers
421 views

Replacing counter in a filename for all files in a directory

After importing several 1000 Files from a camera onto a hard drive I realized, that the counter, used in the process of renaming the file - does not start from 0. This leads to file structure like ...
user avatar
  • 171
1 vote
1 answer
220 views

Recursive (batch) video codec details with MediaInfo CLI

I want to share my script to do this with Media Info CLI and python. At first I tried with pure bash but should have just gone python at first, much quicker and adaptable (for me). My task was to ...
user avatar
-1 votes
1 answer
21 views

Sync sudo anthority to all nodes

I want to submit a task that is interpreted by /bin/csh, which only exists in master node. And I have no root permission but only sudo, which is limited in master node. So I can't use sudo apt install ...
user avatar
  • 1
0 votes
0 answers
182 views

Parallel job submission of for loop

I have written for loop and parallelized it with & and limited the threshold to running 3 jobs at one time. Below is my script. I am reserving 32 cores and 256 GB memory through BSUB command. The ...
user avatar
0 votes
0 answers
37 views

Setting folder permissions in a loop

in an older question I described the problem, that iTunes on Mac sets wrong permissions to folders on a Debian fileserver with a ZFS pool, so that it can't write to these folders. After a while I ...
user avatar
0 votes
2 answers
3k views

how to create a bash script, with alternative of "goto" and "labels"

In batch: @echo off :step1 if exist "file1.txt" (GOTO step2) ELSE (GOTO X) :step2 if exist "file2.txt" (GOTO Z) ELSE (GOTO Y) :X run the script from the beginning and file1.txt ...
user avatar
  • 791
0 votes
2 answers
35 views

Batch rename folders appending fixed text and moving part of the name

I have several directories named in the pattern [year] name, like [1900] aaaa, [1901] bbbb, [1902] cccc, and so on. I'd like to append a fixed text - to the beginning of the name and move the [year] ...
user avatar
  • 103
1 vote
1 answer
200 views

batch awk print from multiple input file to multiple output file

I have multiple text files, and I wish to extract specific columns from these files and save them to *_2.txt files. awk '{print $(NF-3), $5}' *.txt > *_2.txt But this command is not working. How ...
user avatar
  • 113
0 votes
1 answer
605 views

I need to convert Windows batch file into shell script. How can I run an .exe file in Linux with using cron? [closed]

I am trying to convert a Windows batch file into bash shell scripting. But I am not able to understand how the following command can be converted. I have used wine to run .exe file in Linux. Is there ...
user avatar
1 vote
1 answer
192 views

Need to convert a batch of wav in flac in different folder

I need to convert a lot of *.wav in different folder, i try to use find and flac to convert all this without missing name of the track in the folder. This is the line I try to use : find ./ -type f -...
user avatar
0 votes
1 answer
436 views

Rename multiple files according using a names-list

I'm trying to rename multiple numbered files according to a list of names. Example: 1.pdf, 2.pdf, …, n.pdf And a file called names.txt, with a value per line: Fabio Joao n-name So we will have 1....
user avatar
  • 41
0 votes
1 answer
98 views

Run all batch files in a folder and sub-folders

I am new using linux. I have a folder with many sub-folders and sub-sub-folders. All batch files have the same name --> R_batch_1_1.bash Usually I use: sbatch R_batch_1_1.bash to run one by one. ...
user avatar
0 votes
1 answer
96 views

SLURM: How do I submit multilple OpenMP parallel codes in a job script

I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.) #!/bin/bash #SBATCH -N 1 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=2 export OMP_NUM_THREADS=...
user avatar
1 vote
1 answer
229 views

How to identify the exact file which throws out the warning during batch conversion of files using obabel in bash?

I am using obabel tool to batch convert a set of SDF files to PDB file. obabel *.sdf -opdb --gen3d -m I have ~50000 files and i am getting totally of 12 warnings in total (warning mentioned below)....
user avatar
0 votes
1 answer
543 views

How to nohup multiple SAS programs at once?

I have several SAS programs test1_2.sas test3_4.sas test7_8.sas and I hope to nohup those scripts at once. I tried to create a shell wrapper such as a test.sh: #test running multiple files ...
user avatar
  • 103
0 votes
1 answer
551 views

Batch mergin mkv video with subtitles using MKVToolNix

For one file I know I can (being my video and sub files in the same directory): mkvmerge -o output-file.mkv --default-track 0 --language 0:es subtitle-file.ass video-file.mkv But how can I do the ...
user avatar
2 votes
1 answer
1k views

ffmpeg merge multiple sets of 2 videos in for loop

I am still pretty new to linux and have been trying very hard at getting this right. Please help me to get this right. I am trying to merge 2 videos (1 from each folder) multiple times in like a ...
user avatar
6 votes
2 answers
778 views

Batch Processing with one task at a time

I'm used to IBM i (AS/400) batch processing with flexible queueing configuration possibilities. I'm searching a similar facility for Linux Batch processing. Most important is that one job at a time ...
user avatar
  • 163
0 votes
1 answer
71 views

How to batch start background tasks?

I have such files: ls mate30pro5g.py Mathematica.py opencv.py I know this command can start a background task: python3 mate30pro5g.py & Now I want to start multiple tasks by this command: ...
user avatar
  • 987
2 votes
2 answers
369 views

queue-like behaviour for multiple one-core jobs on single machine? [duplicate]

I've got an old Mac with 24 cores, and I'd like to run several hundred/thousands one-core jobs automatically. I've made a bash script that runs the processes in the background, but if I set too many ...
user avatar
  • 23
1 vote
1 answer
619 views

sftp with batch mode suppresses mput and mget command response echo

I recently upgraded OpenSSH from 6.0 to 7.5. My code captures the put/get command response echo and processes it for verification. Code: exec 3>&1 4>&2 exec > $TempSessionLog ...
user avatar
  • 11
0 votes
1 answer
499 views

Batch download a series of files and add prefix to them

So I have a list of downloadable links with a specific order, but the filenames are jumbled up example.com/ddk.zip example.com/453.zip example.com/hth.zip So I want to automatically prefix them as I ...
user avatar
  • 3
1 vote
2 answers
69 views

What's the best way to run a long script without the SSH client crashing?

I'm on OpenSUSE 12.1, so no tmux, and we're not allowed to install anything - wget is too old to download a binary as well. Often I and other users have to run long scripts that take several hours, ...
user avatar
0 votes
1 answer
200 views

batch rename files in Ubuntu: sequential numbering, based on order in directory. (Ubuntu 16.04)

I had to download videos from DVR in more trials. After that files were manually arranged by alphabetical order into few groups, in each group by numbers. It is necessary to have videos in former ...
user avatar
1 vote
1 answer
140 views

Starting batch jobs at exact time slightly before the new hour starts

I've been reading the recent blogpost "Winding down my Debian involvement" by Michael Stapelberg. Sad details aside, it's been mentioned that within Debian infrastructure batch jobs run four times a ...
user avatar
  • 305
1 vote
1 answer
1k views

How do I delete a single file without affecting the marked files in ranger?

As a follow up to this question, how do I delete a single file without losing the selection that I've already got? for i in $(seq 1 5); do touch "$i.jpg"; done; mkdir subdir Let's say you marked 1....
user avatar
  • 22.8k
0 votes
0 answers
42 views

Batch moving a custom set of images with Ranger? [duplicate]

This is a question for the Ranger (file manager) Let's say that I have a directory with five files.. for i in $(seq 1 5); do touch "$i.jpg"; done; mkdir subdir How would I move 1.jpg, and 4.jpg, ...
user avatar
  • 22.8k
0 votes
1 answer
188 views

How to run the same command to execute a file in multiple directories?

I want to do the following, I have a set of directories: e.g) 400K 500K 600K and so on In each directory I have a "run.pbs" file, that I want to submit through batch with "qsub run.pbs". I was ...
user avatar
1 vote
1 answer
368 views

batch rename folders with value from json value in package.json

This is a bit of a crazy one, but I am wondering if it is even possible. I have maybe 300 folders in my /var/www folder that were all renamed to the same name [Name](Num) so kindred (19) for example. ...
user avatar
1 vote
1 answer
1k views

batch script run command if available memory less than X mb

I would like write a batch script that checks used or available memory allow me to run commands if available memory less than X mb. I googled but page they refer didn't work for me I am using centos ...
user avatar
  • 304
0 votes
3 answers
234 views

Selectively batch copy files

I've got a directory containing these files: RG000909-A1_1.jpg RG000909-A2_1.jpg RG000909-A3_1.jpg RG000909-A4_1.jpg RG000909-A5_1.jpg RG000909-A6_1.jpg RG000909-A7_1.jpg RG000909-A8_1.jpg ...
user avatar
0 votes
1 answer
2k views

How can I download and save all the images shown in a web page?

I want to download the images shown in a web page, for example https://www.wikiart.org/en/paintings-by-style/art-nouveau-modern, (the ones under the All Artworks tab), and store them in a folder (e.g. ...
user avatar
  • 1
4 votes
0 answers
936 views

Distinguish between error and "success" in scanimage-batch

I'm running a little script with the scanimage batch-command on a remote server and would like to know if and how the scan has been done the batch. Therefore the script requires a proper "error"-...
user avatar
  • 437
1 vote
0 answers
349 views

How to Sequentially batch sas codes in queue?

I have couple of SAS jobs that I need to run sequentially and some in parallel. I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few ...
user avatar
1 vote
2 answers
2k views

Count unique lines only to a set pattern

How can I count the unique log lines in a text file only until the first "-" and print the line with the count org.springframework. - initialization started org.springframework. - initialization ...
user avatar
2 votes
2 answers
82 views

Batch conversion via filelist.csv

I need to batch convert some video files from H264 to H265. I know my ffmpeg command line, I have a csv of file names with a relative path (I need the filelist to drive it as some files in the ...
user avatar
  • 21
3 votes
0 answers
1k views

How to reliably detect stdin pipe in Bash in batch mode?

I need a way to test whether input to my Bash script has been piped to stdin when the script is run in batch mode. The method I use when I run the script interactively (the test [ -p /dev/stdin ]) ...
user avatar
1 vote
0 answers
33 views

Create a file at time 't' after PBS job execution begins

I submit a PBS job for 02:00:00 hours. I need to create a file in the PBS working directory at a specified time t (say 01:30:00 hours) after the job has begun to ensure that the job terminates ...
user avatar
0 votes
1 answer
132 views

Closing a terminal from an app that was started in that terminal

At work we have a supercomputer accessible via SSH. We login, reserve the cores using bsub command with parameters, and the latter creates (from doc) "an interactive bash session with pseudo-terminal....
user avatar
  • 136
3 votes
1 answer
5k views

How to set the path of a batch environment to find the location of the "qsub" command

I have a program, called "minvappmpi", which is parallelised and itself calls qsub as a subroutine. When I use the program on the cluster, it throws the error qsub: command not found If I want to ...
user avatar
1 vote
1 answer
883 views

Increase allocated nodes for queued jobs on a PBS cluster

I have submitted a job on a PBS managed cluster using qsub -l nodes=5:ppn=16. My job is queued and is waiting for other jobs to be completed. Is there a command that allows me change the number of ...
user avatar
  • 177
1 vote
1 answer
185 views

Rename command to delete substring

I am trying to batch rename the following files: art-faculty-3_29060055362_o.jpeg fine-arts-division-faculty-2016-2017-5_29165851925_o.jpeg theatre-faculty-2016-2017-1_29132529356_o.jpeg art-...
user avatar
  • 13
2 votes
1 answer
4k views

How to run batch sas job in unix sas?

I have 5 SAS jobs that I need to run sequentially, one after the other. I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few hours. ...
user avatar
6 votes
4 answers
2k views

atd, batch // Setting the load limiting factor

I am launching non interactive jobs using batch, and I would like to increase the load limiting factor in order to use all 8 of my cores. I am on Ubuntu 16.04 LTS. From what I understand, batch uses ...
user avatar
  • 81
-1 votes
2 answers
741 views

Job Priority in Batch Jobs

I do not have administrative privileges on the cluster that i am using. I'd like to prioritize some of the jobs that I have submitted. How do i incorporate that into my batch file: #!/bin/bash #$ -...
user avatar