Questions tagged [batch-jobs]
Grid engine batch job processing (qsub, qstat, …)
69
questions
0
votes
0
answers
47
views
How to use ${LSB_JOB_INDEX} to run a for loop in LSF platform
I have a for loop that looks like this, and I want to convert it to an array job.
I want to run 100 jobs in parallel, but I am not entirely sure where I can add
the ${LSB_JOB_INDEX} parameter.
Any ...
- 101
0
votes
1
answer
248
views
Batch writing text to speech audio files from .csv file
I am new to using ZSH on Mac and am looking for help to write a simple shell script to batch create WAV audio files from a .csv file
I have created a .csv file with data in 3 columns, A B C. Wanting ...
- 1
0
votes
2
answers
533
views
ZSh script to batch process filenames containing spaces and substitute extension
Is there a zsh script that can convert a directory files in one music format and be converted into Ogg opus format. The filenames have spaces in their names.
For example, a directory contains 10 ...
- 101
0
votes
1
answer
730
views
zsh: bad subscript for direct array assignment: 0
I have the following bashscript below. My goal is to itterate over multiple files in the directory. The name of the files will be batch_1, batch_2, batch_3, batch_4, etc. There should be no more than ...
- 1
0
votes
0
answers
50
views
In middle of the bash script next job/code takes time to trigger if first is completed
I have a script which have my codes:
# scheduling details for abc.sas, first status check after 1 hr , status check interval is 30 mins
execute "claimant.sas"
echo "execute function ...
6
votes
4
answers
831
views
Replacing counter in a filename for all files in a directory
After importing several 1000 Files from a camera onto a hard drive I realized, that the counter, used in the process of renaming the file - does not start from 0. This leads to file structure like ...
- 171
1
vote
1
answer
647
views
Recursive (batch) video codec details with MediaInfo CLI
I want to share my script to do this with Media Info CLI and python. At first I tried with pure bash but should have just gone python at first, much quicker and adaptable (for me).
My task was to ...
- 252
-1
votes
1
answer
29
views
Sync sudo anthority to all nodes
I want to submit a task that is interpreted by /bin/csh, which only exists in master node. And I have no root permission but only sudo, which is limited in master node. So I can't use sudo apt install ...
- 1
1
vote
0
answers
743
views
Parallel job submission of for loop
I have written for loop and parallelized it with & and limited the threshold to running 3 jobs at one time. Below is my script. I am reserving 32 cores and 256 GB memory through BSUB command. The ...
- 137
1
vote
2
answers
11k
views
how to create a bash script, with alternative of "goto" and "labels"
In batch:
@echo off
:step1
if exist "file1.txt" (GOTO step2) ELSE (GOTO X)
:step2
if exist "file2.txt" (GOTO Z) ELSE (GOTO Y)
:X
run the script from the beginning and file1.txt ...
- 891
0
votes
2
answers
56
views
Batch rename folders appending fixed text and moving part of the name
I have several directories named in the pattern [year] name, like [1900] aaaa, [1901] bbbb, [1902] cccc, and so on. I'd like to append a fixed text - to the beginning of the name and move the [year] ...
- 103
1
vote
1
answer
452
views
batch awk print from multiple input file to multiple output file
I have multiple text files, and I wish to extract specific columns from these files and save them to *_2.txt files.
awk '{print $(NF-3), $5}' *.txt > *_2.txt
But this command is not working. How ...
- 113
0
votes
1
answer
2k
views
I need to convert Windows batch file into shell script. How can I run an .exe file in Linux with using cron? [closed]
I am trying to convert a Windows batch file into bash shell scripting. But I am not able to understand how the following command can be converted. I have used wine to run .exe file in Linux. Is there ...
1
vote
1
answer
484
views
Need to convert a batch of wav in flac in different folder
I need to convert a lot of *.wav in different folder, i try to use find and flac to convert all this without missing name of the track in the folder.
This is the line I try to use :
find ./ -type f -...
- 13
1
vote
1
answer
1k
views
Rename multiple files according using a names-list
I'm trying to rename multiple numbered files according to a list of names.
Example:
1.pdf, 2.pdf, …, n.pdf
And a file called names.txt, with a value per line:
Fabio
Joao
n-name
So we will have
1....
- 43
0
votes
1
answer
331
views
Run all batch files in a folder and sub-folders
I am new using linux.
I have a folder with many sub-folders and sub-sub-folders.
All batch files have the same name --> R_batch_1_1.bash
Usually I use:
sbatch R_batch_1_1.bash to run one by one.
...
- 1
0
votes
1
answer
291
views
SLURM: How do I submit multilple OpenMP parallel codes in a job script
I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.)
#!/bin/bash
#SBATCH -N 1
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=2
export OMP_NUM_THREADS=...
- 103
1
vote
1
answer
362
views
How to identify the exact file which throws out the warning during batch conversion of files using obabel in bash?
I am using obabel tool to batch convert a set of SDF files to PDB file.
obabel *.sdf -opdb --gen3d -m
I have ~50000 files and i am getting totally of 12 warnings in total (warning mentioned below)....
- 584
0
votes
1
answer
938
views
How to nohup multiple SAS programs at once?
I have several SAS programs
test1_2.sas
test3_4.sas
test7_8.sas
and I hope to nohup those scripts at once. I tried to create a shell wrapper such as a test.sh:
#test running multiple files
...
- 117
0
votes
1
answer
993
views
Batch mergin mkv video with subtitles using MKVToolNix
For one file I know I can (being my video and sub files in the same directory):
mkvmerge -o output-file.mkv --default-track 0 --language 0:es subtitle-file.ass video-file.mkv
But how can I do the ...
- 157
2
votes
1
answer
2k
views
ffmpeg merge multiple sets of 2 videos in for loop
I am still pretty new to linux and have been trying very hard at getting this right. Please help me to get this right.
I am trying to merge 2 videos (1 from each folder) multiple times in like a ...
- 33
6
votes
2
answers
1k
views
Batch Processing with one task at a time
I'm used to IBM i (AS/400) batch processing with flexible queueing configuration possibilities. I'm searching a similar facility for Linux Batch processing.
Most important is that one job at a time ...
- 163
0
votes
1
answer
126
views
How to batch start background tasks?
I have such files:
ls
mate30pro5g.py Mathematica.py opencv.py
I know this command can start a background task:
python3 mate30pro5g.py &
Now I want to start multiple tasks by this command:
...
- 1,037
2
votes
2
answers
524
views
queue-like behaviour for multiple one-core jobs on single machine? [duplicate]
I've got an old Mac with 24 cores, and I'd like to run several hundred/thousands one-core jobs automatically. I've made a bash script that runs the processes in the background, but if I set too many ...
- 23
1
vote
1
answer
1k
views
sftp with batch mode suppresses mput and mget command response echo
I recently upgraded OpenSSH from 6.0 to 7.5.
My code captures the put/get command response echo and processes it for verification.
Code:
exec 3>&1 4>&2
exec > $TempSessionLog ...
- 11
0
votes
1
answer
808
views
Batch download a series of files and add prefix to them
So I have a list of downloadable links with a specific order, but the filenames are jumbled up
example.com/ddk.zip
example.com/453.zip
example.com/hth.zip
So I want to automatically prefix them as I ...
- 3
1
vote
2
answers
84
views
What's the best way to run a long script without the SSH client crashing?
I'm on OpenSUSE 12.1, so no tmux, and we're not allowed to install anything - wget is too old to download a binary as well. Often I and other users have to run long scripts that take several hours, ...
user351959
0
votes
1
answer
279
views
batch rename files in Ubuntu: sequential numbering, based on order in directory. (Ubuntu 16.04)
I had to download videos from DVR in more trials. After that files were manually arranged by alphabetical order into few groups, in each group by numbers. It is necessary to have videos in former ...
1
vote
1
answer
167
views
Starting batch jobs at exact time slightly before the new hour starts
I've been reading the recent blogpost "Winding down my Debian involvement" by Michael Stapelberg.
Sad details aside, it's been mentioned that within Debian infrastructure batch jobs run four times a ...
- 305
1
vote
1
answer
2k
views
How do I delete a single file without affecting the marked files in ranger?
As a follow up to this question, how do I delete a single file without losing the selection that I've already got?
for i in $(seq 1 5); do touch "$i.jpg"; done;
mkdir subdir
Let's say you marked 1....
- 29k
0
votes
0
answers
43
views
Batch moving a custom set of images with Ranger? [duplicate]
This is a question for the Ranger (file manager)
Let's say that I have a directory with five files..
for i in $(seq 1 5); do touch "$i.jpg"; done;
mkdir subdir
How would I move 1.jpg, and 4.jpg, ...
- 29k
0
votes
1
answer
310
views
How to run the same command to execute a file in multiple directories?
I want to do the following, I have a set of directories:
e.g) 400K 500K 600K and so on
In each directory I have a "run.pbs" file, that I want to submit through batch with "qsub run.pbs".
I was ...
1
vote
1
answer
550
views
batch rename folders with value from json value in package.json
This is a bit of a crazy one, but I am wondering if it is even possible.
I have maybe 300 folders in my /var/www folder that were all renamed to the same name [Name](Num) so kindred (19) for example.
...
- 121
1
vote
1
answer
1k
views
batch script run command if available memory less than X mb
I would like write a batch script that checks used or available memory allow me to run commands if available memory less than X mb.
I googled but page they refer didn't work for me I am using centos ...
- 304
0
votes
3
answers
510
views
Selectively batch copy files
I've got a directory containing these files:
RG000909-A1_1.jpg
RG000909-A2_1.jpg
RG000909-A3_1.jpg
RG000909-A4_1.jpg
RG000909-A5_1.jpg
RG000909-A6_1.jpg
RG000909-A7_1.jpg
RG000909-A8_1.jpg
...
- 75
0
votes
1
answer
3k
views
How can I download and save all the images shown in a web page?
I want to download the images shown in a web page, for example https://www.wikiart.org/en/paintings-by-style/art-nouveau-modern, (the ones under the All Artworks tab), and store them in a folder (e.g. ...
- 1
4
votes
0
answers
1k
views
Distinguish between error and "success" in scanimage-batch
I'm running a little script with the scanimage batch-command on a remote server and would like to know if and how the scan has been done the batch. Therefore the script requires a proper "error"-...
- 507
1
vote
0
answers
446
views
How to Sequentially batch sas codes in queue?
I have couple of SAS jobs that I need to run sequentially and some in parallel.
I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few ...
- 11
1
vote
2
answers
2k
views
Count unique lines only to a set pattern
How can I count the unique log lines in a text file only until the first "-" and print the line with the count
org.springframework. - initialization started
org.springframework. - initialization ...
- 13
2
votes
2
answers
130
views
Batch conversion via filelist.csv
I need to batch convert some video files from H264 to H265. I know my ffmpeg command line, I have a csv of file names with a relative path (I need the filelist to drive it as some files in the ...
- 21
3
votes
0
answers
1k
views
How to reliably detect stdin pipe in Bash in batch mode?
I need a way to test whether input to my Bash script has been piped
to stdin when the script is run in batch mode. The method I use when
I run the script interactively (the test [ -p /dev/stdin ]) ...
- 131
1
vote
0
answers
42
views
Create a file at time 't' after PBS job execution begins
I submit a PBS job for 02:00:00 hours. I need to create a file in the PBS working directory at a specified time t (say 01:30:00 hours) after the job has begun to ensure that the job terminates ...
- 435
0
votes
1
answer
199
views
Closing a terminal from an app that was started in that terminal
At work we have a supercomputer accessible via SSH.
We login, reserve the cores using bsub command with parameters, and the latter creates (from doc) "an interactive bash session with pseudo-terminal....
- 146
3
votes
1
answer
5k
views
How to set the path of a batch environment to find the location of the "qsub" command
I have a program, called "minvappmpi", which is parallelised and itself calls qsub as a subroutine.
When I use the program on the cluster, it throws the error
qsub: command not found
If I want to ...
- 61
1
vote
1
answer
1k
views
Increase allocated nodes for queued jobs on a PBS cluster
I have submitted a job on a PBS managed cluster using qsub -l nodes=5:ppn=16. My job is queued and is waiting for other jobs to be completed.
Is there a command that allows me change the number of ...
- 307
1
vote
1
answer
204
views
Rename command to delete substring
I am trying to batch rename the following files:
art-faculty-3_29060055362_o.jpeg
fine-arts-division-faculty-2016-2017-5_29165851925_o.jpeg
theatre-faculty-2016-2017-1_29132529356_o.jpeg
art-...
- 13
3
votes
1
answer
5k
views
How to run batch sas job in unix sas?
I have 5 SAS jobs that I need to run sequentially, one after the other.
I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few hours.
...
- 81
6
votes
4
answers
2k
views
atd, batch // Setting the load limiting factor
I am launching non interactive jobs using batch, and I would like to increase the load limiting factor in order to use all 8 of my cores. I am on Ubuntu 16.04 LTS.
From what I understand, batch uses ...
- 81
-1
votes
2
answers
854
views
Job Priority in Batch Jobs
I do not have administrative privileges on the cluster that i am using. I'd like to prioritize some of the jobs that I have submitted. How do i incorporate that into my batch file:
#!/bin/bash
#$ -...
3
votes
1
answer
376
views
filter document via aspell
I need to get a clean txt document and my first approach is to use aspell. The issue is I need it on batch, no interactive mode. Every txt file is piped to aspell and must be returned a new document ...
- 521