Questions tagged [batch-jobs]

Grid engine batch job processing (qsub, qstat, …)

Filter by
Sorted by
Tagged with
1
vote
1answer
47 views

batch awk print from multiple input file to multiple output file

I have multiple text files, and I wish to extract specific columns from these files and save them to *_2.txt files. awk '{print $(NF-3), $5}' *.txt > *_2.txt But this command is not working. How ...
0
votes
1answer
58 views

I need to convert Windows batch file into shell script. How can I run an .exe file in Linux with using cron? [closed]

I am trying to convert a Windows batch file into bash shell scripting. But I am not able to understand how the following command can be converted. I have used wine to run .exe file in Linux. Is there ...
1
vote
1answer
15 views

Need to convert a batch of wav in flac in different folder

I need to convert a lot of *.wav in different folder, i try to use find and flac to convert all this without missing name of the track in the folder. This is the line I try to use : find ./ -type f -...
0
votes
0answers
12 views

TOP/FREE into log file

I'm running Centos 7 and I want to store TOP command into a file ( LOG ) during 24 hours or more. I have something strange running in the VPS on OPENVZ container with CWPanel which have 16 CPU Cores ...
0
votes
1answer
78 views

Rename multiple files according using a names-list

I'm trying to rename multiple numbered files according to a list of names. Example: 1.pdf, 2.pdf, …, n.pdf And a file called names.txt, with a value per line: Fabio Joao n-name So we will have 1....
0
votes
1answer
28 views

Run all batch files in a folder and sub-folders

I am new using linux. I have a folder with many sub-folders and sub-sub-folders. All batch files have the same name --> R_batch_1_1.bash Usually I use: sbatch R_batch_1_1.bash to run one by one. ...
0
votes
1answer
15 views

SLURM: How do I submit multilple OpenMP parallel codes in a job script

I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.) #!/bin/bash #SBATCH -N 1 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=2 export OMP_NUM_THREADS=...
0
votes
0answers
20 views

Download Playlist URLs only on YouTubeDL

I had tried to use YouTubeDL to download all the URL showcases of a Vimeo channel. The use case would be to use on an static website as posting the URLs to forward to Vimeo if the user wants to see an ...
1
vote
1answer
67 views

How to identify the exact file which throws out the warning during batch conversion of files using obabel in bash?

I am using obabel tool to batch convert a set of SDF files to PDB file. obabel *.sdf -opdb --gen3d -m I have ~50000 files and i am getting totally of 12 warnings in total (warning mentioned below)....
0
votes
0answers
146 views

Copy files whose extensions has format of .xxx.xxxxxxxxxx

I have a script that (robo)copies files from different folders, but this batch script copies ALL files for each of the folders. Please see below batch script. I have files in the source folders whose ...
0
votes
1answer
190 views

How to nohup multiple SAS programs at once?

I have several SAS programs test1_2.sas test3_4.sas test7_8.sas and I hope to nohup those scripts at once. I tried to create a shell wrapper such as a test.sh: #test running multiple files ...
0
votes
1answer
89 views

Batch mergin mkv video with subtitles using MKVToolNix

For one file I know I can (being my video and sub files in the same directory): mkvmerge -o output-file.mkv --default-track 0 --language 0:es subtitle-file.ass video-file.mkv But how can I do the ...
0
votes
0answers
44 views

Dependency between outputs in the slurm job

I want to submit a slurm job below in a cluster. Since I have many samples, I want to loop through the samples to submit one job per sample to the cluster. The script writes out two outputs, output1 ...
1
vote
1answer
750 views

ffmpeg merge multiple sets of 2 videos in for loop

I am still pretty new to linux and have been trying very hard at getting this right. Please help me to get this right. I am trying to merge 2 videos (1 from each folder) multiple times in like a ...
6
votes
2answers
271 views

Batch Processing with one task at a time

I'm used to IBM i (AS/400) batch processing with flexible queueing configuration possibilities. I'm searching a similar facility for Linux Batch processing. Most important is that one job at a time ...
0
votes
1answer
51 views

How to batch start background tasks?

I have such files: ls mate30pro5g.py Mathematica.py opencv.py I know this command can start a background task: python3 mate30pro5g.py & Now I want to start multiple tasks by this command: ...
2
votes
2answers
154 views

queue-like behaviour for multiple one-core jobs on single machine? [duplicate]

I've got an old Mac with 24 cores, and I'd like to run several hundred/thousands one-core jobs automatically. I've made a bash script that runs the processes in the background, but if I set too many ...
1
vote
1answer
268 views

sftp with batch mode suppresses mput and mget command response echo

I recently upgraded OpenSSH from 6.0 to 7.5. My code captures the put/get command response echo and processes it for verification. Code: exec 3>&1 4>&2 exec > $TempSessionLog ...
0
votes
1answer
230 views

Batch download a series of files and add prefix to them

So I have a list of downloadable links with a specific order, but the filenames are jumbled up example.com/ddk.zip example.com/453.zip example.com/hth.zip So I want to automatically prefix them as I ...
1
vote
2answers
42 views

What's the best way to run a long script without the SSH client crashing?

I'm on OpenSUSE 12.1, so no tmux, and we're not allowed to install anything - wget is too old to download a binary as well. Often I and other users have to run long scripts that take several hours, ...
0
votes
1answer
102 views

batch rename files in Ubuntu: sequential numbering, based on order in directory. (Ubuntu 16.04)

I had to download videos from DVR in more trials. After that files were manually arranged by alphabetical order into few groups, in each group by numbers. It is necessary to have videos in former ...
1
vote
1answer
106 views

Starting batch jobs at exact time slightly before the new hour starts

I've been reading the recent blogpost "Winding down my Debian involvement" by Michael Stapelberg. Sad details aside, it's been mentioned that within Debian infrastructure batch jobs run four times a ...
1
vote
1answer
515 views

How do I delete a single file without affecting the marked files in ranger?

As a follow up to this question, how do I delete a single file without losing the selection that I've already got? for i in $(seq 1 5); do touch "$i.jpg"; done; mkdir subdir Let's say you marked 1....
0
votes
0answers
39 views

Batch moving a custom set of images with Ranger? [duplicate]

This is a question for the Ranger (file manager) Let's say that I have a directory with five files.. for i in $(seq 1 5); do touch "$i.jpg"; done; mkdir subdir How would I move 1.jpg, and 4.jpg, ...
0
votes
1answer
93 views

How to run the same command to execute a file in multiple directories?

I want to do the following, I have a set of directories: e.g) 400K 500K 600K and so on In each directory I have a "run.pbs" file, that I want to submit through batch with "qsub run.pbs". I was ...
-1
votes
1answer
3k views

Copy file from Windows to Linux WHEN A SHELL SCRIPT IS EXECUTED

I have a program on Linux server, which runs a shell script on Linux server. Say "/tmp/ShellScript.sh" Now I want to copy a file placed on my Windows Machine location "C:\WindowsFolder\FileToCopy" to ...
1
vote
1answer
167 views

batch rename folders with value from json value in package.json

This is a bit of a crazy one, but I am wondering if it is even possible. I have maybe 300 folders in my /var/www folder that were all renamed to the same name [Name](Num) so kindred (19) for example. ...
1
vote
1answer
675 views

batch script run command if available memory less than X mb

I would like write a batch script that checks used or available memory allow me to run commands if available memory less than X mb. I googled but page they refer didn't work for me I am using centos ...
0
votes
3answers
108 views

Selectively batch copy files

I've got a directory containing these files: RG000909-A1_1.jpg RG000909-A2_1.jpg RG000909-A3_1.jpg RG000909-A4_1.jpg RG000909-A5_1.jpg RG000909-A6_1.jpg RG000909-A7_1.jpg RG000909-A8_1.jpg ...
0
votes
1answer
922 views

How can I download and save all the images shown in a web page?

I want to download the images shown in a web page, for example https://www.wikiart.org/en/paintings-by-style/art-nouveau-modern, (the ones under the All Artworks tab), and store them in a folder (e.g. ...
4
votes
0answers
635 views

Distinguish between error and “success” in scanimage-batch

I'm running a little script with the scanimage batch-command on a remote server and would like to know if and how the scan has been done the batch. Therefore the script requires a proper "error"-...
1
vote
0answers
289 views

How to Sequentially batch sas codes in queue?

I have couple of SAS jobs that I need to run sequentially and some in parallel. I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few ...
1
vote
2answers
2k views

Count unique lines only to a set pattern

How can I count the unique log lines in a text file only until the first "-" and print the line with the count org.springframework. - initialization started org.springframework. - initialization ...
2
votes
2answers
53 views

Batch conversion via filelist.csv

I need to batch convert some video files from H264 to H265. I know my ffmpeg command line, I have a csv of file names with a relative path (I need the filelist to drive it as some files in the ...
3
votes
0answers
903 views

How to reliably detect stdin pipe in Bash in batch mode?

I need a way to test whether input to my Bash script has been piped to stdin when the script is run in batch mode. The method I use when I run the script interactively (the test [ -p /dev/stdin ]) ...
1
vote
0answers
31 views

Create a file at time 't' after PBS job execution begins

I submit a PBS job for 02:00:00 hours. I need to create a file in the PBS working directory at a specified time t (say 01:30:00 hours) after the job has begun to ensure that the job terminates ...
0
votes
1answer
98 views

Closing a terminal from an app that was started in that terminal

At work we have a supercomputer accessible via SSH. We login, reserve the cores using bsub command with parameters, and the latter creates (from doc) "an interactive bash session with pseudo-terminal....
2
votes
1answer
4k views

How to set the path of a batch environment to find the location of the “qsub” command

I have a program, called "minvappmpi", which is parallelised and itself calls qsub as a subroutine. When I use the program on the cluster, it throws the error qsub: command not found If I want to ...
1
vote
1answer
729 views

Increase allocated nodes for queued jobs on a PBS cluster

I have submitted a job on a PBS managed cluster using qsub -l nodes=5:ppn=16. My job is queued and is waiting for other jobs to be completed. Is there a command that allows me change the number of ...
1
vote
1answer
155 views

Rename command to delete substring

I am trying to batch rename the following files: art-faculty-3_29060055362_o.jpeg fine-arts-division-faculty-2016-2017-5_29165851925_o.jpeg theatre-faculty-2016-2017-1_29132529356_o.jpeg art-...
2
votes
1answer
3k views

How to run batch sas job in unix sas?

I have 5 SAS jobs that I need to run sequentially, one after the other. I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few hours. ...
6
votes
3answers
1k views

atd, batch // Setting the load limiting factor

I am launching non interactive jobs using batch, and I would like to increase the load limiting factor in order to use all 8 of my cores. I am on Ubuntu 16.04 LTS. From what I understand, batch uses ...
0
votes
2answers
613 views

Job Priority in Batch Jobs

I do not have administrative privileges on the cluster that i am using. I'd like to prioritize some of the jobs that I have submitted. How do i incorporate that into my batch file: #!/bin/bash #$ -...
3
votes
1answer
294 views

filter document via aspell

I need to get a clean txt document and my first approach is to use aspell. The issue is I need it on batch, no interactive mode. Every txt file is piped to aspell and must be returned a new document ...
1
vote
1answer
97 views

submitting jobs to get 3 nodes running parallel executions

I have a submit script looks like below, it tries to run a large number of instances of csce.py in backgrounds with 3 nodes.... in a laptop, this usually could successfully automatically distribute ...
0
votes
1answer
604 views

How to make a batch file answer a user prompt

I have a batch file that runs a command, and one of those commands requires you to type in yes and hit enter. Is there any way that the batch file can do this?
1
vote
2answers
178 views

Batch rename long filenames of variable length with years

I have a bunch of files in a single directory that I would like to rename as follows, example existing names: 1937 - Snow White and the Seven Dwarves.avi 1940 - Pinocchio.avi Target names: Snow ...
0
votes
1answer
28 views

rename all files in a folder deleting duplicate string-parts

I have a folder with lots of subfolders, that contain two times the same but slightly different: Movie1 {Action}{Adventure}{Sci-Fi}{Thriller}{Science Fiction}/ Movie2 {Action}{Adventure}{Thriller}{...
77
votes
7answers
148k views

Why is scp so slow and how to make it faster?

I'm trying to copy a batch of files with scp but it is very slow. This is an example with 10 files: $ time scp cap_* user@host:~/dir cap_20151023T113018_704979707.png 100% 413KB 413.2KB/s 00:00 ...
2
votes
0answers
542 views

Need a script for automation the convert a lot number audio files to another format [closed]

I have a lot number audio files in MP3 the proprietary format, I want to convert them to 'opus' the free and high quality format, with keep metadata also. My selection command-line programs are SoX (...