Questions tagged [batch-jobs]

Grid engine batch job processing (qsub, qstat, …)

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

SLURM: How do I submit multilple OpenMP parallel codes in a job script

I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.) #!/bin/bash #SBATCH -N 1 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=2 export OMP_NUM_THREADS=...
0
votes
0answers
14 views

Download Playlist URLs only on YouTubeDL

I had tried to use YouTubeDL to download all the URL showcases of a Vimeo channel. The use case would be to use on an static website as posting the URLs to forward to Vimeo if the user wants to see an ...
1
vote
1answer
40 views

How to identify the exact file which throws out the warning during batch conversion of files using obabel in bash?

I am using obabel tool to batch convert a set of SDF files to PDB file. obabel *.sdf -opdb --gen3d -m I have ~50000 files and i am getting totally of 12 warnings in total (warning mentioned below)....
0
votes
0answers
105 views

Copy files whose extensions has format of .xxx.xxxxxxxxxx

I have a script that (robo)copies files from different folders, but this batch script copies ALL files for each of the folders. Please see below batch script. I have files in the source folders whose ...
0
votes
1answer
76 views

How to nohup multiple SAS programs at once?

I have several SAS programs test1_2.sas test3_4.sas test7_8.sas and I hope to nohup those scripts at once. I tried to create a shell wrapper such as a test.sh: #test running multiple files ...
0
votes
1answer
26 views

Batch mergin mkv video with subtitles using MKVToolNix

For one file I know I can (being my video and sub files in the same directory): mkvmerge -o output-file.mkv --default-track 0 --language 0:es subtitle-file.ass video-file.mkv But how can I do the ...
0
votes
0answers
30 views

Dependency between outputs in the slurm job

I want to submit a slurm job below in a cluster. Since I have many samples, I want to loop through the samples to submit one job per sample to the cluster. The script writes out two outputs, output1 ...
1
vote
1answer
365 views

ffmpeg merge multiple sets of 2 videos in for loop

I am still pretty new to linux and have been trying very hard at getting this right. Please help me to get this right. I am trying to merge 2 videos (1 from each folder) multiple times in like a ...
6
votes
2answers
145 views

Batch Processing with one task at a time

I'm used to IBM i (AS/400) batch processing with flexible queueing configuration possibilities. I'm searching a similar facility for Linux Batch processing. Most important is that one job at a time ...
0
votes
1answer
46 views

How to batch start background tasks?

I have such files: ls mate30pro5g.py Mathematica.py opencv.py I know this command can start a background task: python3 mate30pro5g.py & Now I want to start multiple tasks by this command: ...
2
votes
2answers
88 views

queue-like behaviour for multiple one-core jobs on single machine? [duplicate]

I've got an old Mac with 24 cores, and I'd like to run several hundred/thousands one-core jobs automatically. I've made a bash script that runs the processes in the background, but if I set too many ...
1
vote
1answer
179 views

sftp with batch mode suppresses mput and mget command response echo

I recently upgraded OpenSSH from 6.0 to 7.5. My code captures the put/get command response echo and processes it for verification. Code: exec 3>&1 4>&2 exec > $TempSessionLog ...
0
votes
1answer
141 views

Batch download a series of files and add prefix to them

So I have a list of downloadable links with a specific order, but the filenames are jumbled up example.com/ddk.zip example.com/453.zip example.com/hth.zip So I want to automatically prefix them as I ...
1
vote
2answers
41 views

What's the best way to run a long script without the SSH client crashing?

I'm on OpenSUSE 12.1, so no tmux, and we're not allowed to install anything - wget is too old to download a binary as well. Often I and other users have to run long scripts that take several hours, ...
0
votes
1answer
85 views

batch rename files in Ubuntu: sequential numbering, based on order in directory. (Ubuntu 16.04)

I had to download videos from DVR in more trials. After that files were manually arranged by alphabetical order into few groups, in each group by numbers. It is necessary to have videos in former ...
1
vote
1answer
94 views

Starting batch jobs at exact time slightly before the new hour starts

I've been reading the recent blogpost "Winding down my Debian involvement" by Michael Stapelberg. Sad details aside, it's been mentioned that within Debian infrastructure batch jobs run four times a ...
1
vote
0answers
325 views

How do I delete a single one-off file without disturbing the batch-job I'm making?

As a follow up to this question, how do I delete a single file without losing the selection that I've already got? for i in $(seq 1 5); do touch "$i.jpg"; done; mkdir subdir Let's say you marked 1....
0
votes
0answers
32 views

Batch moving a custom set of images with Ranger? [duplicate]

This is a question for the Ranger (file manager) Let's say that I have a directory with five files.. for i in $(seq 1 5); do touch "$i.jpg"; done; mkdir subdir How would I move 1.jpg, and 4.jpg, ...
0
votes
1answer
87 views

How to run the same command to execute a file in multiple directories?

I want to do the following, I have a set of directories: e.g) 400K 500K 600K and so on In each directory I have a "run.pbs" file, that I want to submit through batch with "qsub run.pbs". I was ...
-1
votes
1answer
2k views

Copy file from Windows to Linux WHEN A SHELL SCRIPT IS EXECUTED

I have a program on Linux server, which runs a shell script on Linux server. Say "/tmp/ShellScript.sh" Now I want to copy a file placed on my Windows Machine location "C:\WindowsFolder\FileToCopy" to ...
1
vote
1answer
122 views

batch rename folders with value from json value in package.json

This is a bit of a crazy one, but I am wondering if it is even possible. I have maybe 300 folders in my /var/www folder that were all renamed to the same name [Name](Num) so kindred (19) for example. ...
1
vote
1answer
577 views

batch script run command if available memory less than X mb

I would like write a batch script that checks used or available memory allow me to run commands if available memory less than X mb. I googled but page they refer didn't work for me I am using centos ...
0
votes
3answers
92 views

Selectively batch copy files

I've got a directory containing these files: RG000909-A1_1.jpg RG000909-A2_1.jpg RG000909-A3_1.jpg RG000909-A4_1.jpg RG000909-A5_1.jpg RG000909-A6_1.jpg RG000909-A7_1.jpg RG000909-A8_1.jpg ...
0
votes
1answer
612 views

How can I download and save all the images shown in a web page?

I want to download the images shown in a web page, for example https://www.wikiart.org/en/paintings-by-style/art-nouveau-modern, (the ones under the All Artworks tab), and store them in a folder (e.g. ...
4
votes
0answers
543 views

Distinguish between error and “success” in scanimage-batch

I'm running a little script with the scanimage batch-command on a remote server and would like to know if and how the scan has been done the batch. Therefore the script requires a proper "error"-...
1
vote
0answers
265 views

How to Sequentially batch sas codes in queue?

I have couple of SAS jobs that I need to run sequentially and some in parallel. I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few ...
1
vote
2answers
1k views

Count unique lines only to a set pattern

How can I count the unique log lines in a text file only until the first "-" and print the line with the count org.springframework. - initialization started org.springframework. - initialization ...
2
votes
2answers
50 views

Batch conversion via filelist.csv

I need to batch convert some video files from H264 to H265. I know my ffmpeg command line, I have a csv of file names with a relative path (I need the filelist to drive it as some files in the ...
3
votes
0answers
830 views

How to reliably detect stdin pipe in Bash in batch mode?

I need a way to test whether input to my Bash script has been piped to stdin when the script is run in batch mode. The method I use when I run the script interactively (the test [ -p /dev/stdin ]) ...
1
vote
0answers
31 views

Create a file at time 't' after PBS job execution begins

I submit a PBS job for 02:00:00 hours. I need to create a file in the PBS working directory at a specified time t (say 01:30:00 hours) after the job has begun to ensure that the job terminates ...
0
votes
1answer
90 views

Closing a terminal from an app that was started in that terminal

At work we have a supercomputer accessible via SSH. We login, reserve the cores using bsub command with parameters, and the latter creates (from doc) "an interactive bash session with pseudo-terminal....
2
votes
1answer
3k views

How to set the path of a batch environment to find the location of the “qsub” command

I have a program, called "minvappmpi", which is parallelised and itself calls qsub as a subroutine. When I use the program on the cluster, it throws the error qsub: command not found If I want to ...
1
vote
1answer
674 views

Increase allocated nodes for queued jobs on a PBS cluster

I have submitted a job on a PBS managed cluster using qsub -l nodes=5:ppn=16. My job is queued and is waiting for other jobs to be completed. Is there a command that allows me change the number of ...
1
vote
1answer
148 views

Rename command to delete substring

I am trying to batch rename the following files: art-faculty-3_29060055362_o.jpeg fine-arts-division-faculty-2016-2017-5_29165851925_o.jpeg theatre-faculty-2016-2017-1_29132529356_o.jpeg art-...
2
votes
1answer
3k views

How to run batch sas job in unix sas?

I have 5 SAS jobs that I need to run sequentially, one after the other. I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few hours. ...
6
votes
3answers
1k views

atd, batch // Setting the load limiting factor

I am launching non interactive jobs using batch, and I would like to increase the load limiting factor in order to use all 8 of my cores. I am on Ubuntu 16.04 LTS. From what I understand, batch uses ...
0
votes
2answers
563 views

Job Priority in Batch Jobs

I do not have administrative privileges on the cluster that i am using. I'd like to prioritize some of the jobs that I have submitted. How do i incorporate that into my batch file: #!/bin/bash #$ -...
3
votes
1answer
278 views

filter document via aspell

I need to get a clean txt document and my first approach is to use aspell. The issue is I need it on batch, no interactive mode. Every txt file is piped to aspell and must be returned a new document ...
1
vote
1answer
97 views

submitting jobs to get 3 nodes running parallel executions

I have a submit script looks like below, it tries to run a large number of instances of csce.py in backgrounds with 3 nodes.... in a laptop, this usually could successfully automatically distribute ...
0
votes
1answer
582 views

How to make a batch file answer a user prompt

I have a batch file that runs a command, and one of those commands requires you to type in yes and hit enter. Is there any way that the batch file can do this?
1
vote
2answers
161 views

Batch rename long filenames of variable length with years

I have a bunch of files in a single directory that I would like to rename as follows, example existing names: 1937 - Snow White and the Seven Dwarves.avi 1940 - Pinocchio.avi Target names: Snow ...
0
votes
1answer
28 views

rename all files in a folder deleting duplicate string-parts

I have a folder with lots of subfolders, that contain two times the same but slightly different: Movie1 {Action}{Adventure}{Sci-Fi}{Thriller}{Science Fiction}/ Movie2 {Action}{Adventure}{Thriller}{...
71
votes
7answers
132k views

Why is scp so slow and how to make it faster?

I'm trying to copy a batch of files with scp but it is very slow. This is an example with 10 files: $ time scp cap_* user@host:~/dir cap_20151023T113018_704979707.png 100% 413KB 413.2KB/s 00:00 ...
2
votes
0answers
529 views

Need a script for automation the convert a lot number audio files to another format [closed]

I have a lot number audio files in MP3 the proprietary format, I want to convert them to 'opus' the free and high quality format, with keep metadata also. My selection command-line programs are SoX (...
1
vote
2answers
472 views

multiple files rename - filenames patterns [duplicate]

I have following image files in some directory which I want to be renamed: 8 -rw-rw-r-- 1 6661 sep 24 10:28 dbConnectionOkBostjan.png 8 -rw-rw-r-- 1 6548 sep 24 10:29 dbConnectionErrorBostjan.png 8 -...
2
votes
1answer
239 views

How to have a process queue in AIX like xargs with “--max-procs”?

Consider a simple processing queue like: cat list.txt | xargs -n1 -P20 process.sh (-P or --max-procs) How to have something like that in AIX ?
3
votes
0answers
9k views

qsub job in state Eqw error: can't chdir to directory: No such file or directory

I'm trying to submit a job on a cluster via qsub, but it gets stuck in state "Eqw" with error message: $ qstat -j 466 | grep error error reason 1: 09/18/2015 17:12:32 [1125:3453]: error: ...
11
votes
1answer
8k views

Is it possible to pause jobs submitted via qsub?

I am running my jobs on a small cluster. I submitted them via qsub. Now my labmates need resources more urgently than me, so I need to either kill my jobs or pause them, if possible. Is there a way ...
0
votes
1answer
995 views

How do I run a batch file at a specific time?

I have tried using the batch and at commands with no luck. I have a batch file in a location, lets say "$DD/batchfile.batch". Here is what I have tried. batch -f $DD/batchfile.batch 10:00 today at -f ...
23
votes
6answers
48k views

How to download files and folders from Onedrive using wget?

How to use wget to download files from Onedrive? (and batch files and entire folders, if possible)