Questions tagged [batch-jobs]

Grid engine batch job processing (qsub, qstat, …)

0
votes
1answer
35 views

Batch download a series of files and add prefix to them

So I have a list of downloadable links with a specific order, but the filenames are jumbled up example.com/ddk.zip example.com/453.zip example.com/hth.zip So I want to automatically prefix them as I ...
1
vote
2answers
30 views

What's the best way to run a long script without the SSH client crashing?

I'm on OpenSUSE 12.1, so no tmux, and we're not allowed to install anything - wget is too old to download a binary as well. Often I and other users have to run long scripts that take several hours, ...
0
votes
0answers
32 views

How can I convert all bin-cue files to iso in a folder with bchunk respecting its filenames? (PSX mode)

I know that the command is bchunk -p foo.bin foo.cue foo, but how I could conver all bin/cue files in a folder to iso at once? Thank you.
0
votes
1answer
34 views

batch rename files in Ubuntu: sequential numbering, based on order in directory. (Ubuntu 16.04)

I had to download videos from DVR in more trials. After that files were manually arranged by alphabetical order into few groups, in each group by numbers. It is necessary to have videos in former ...
1
vote
1answer
61 views

Starting batch jobs at exact time slightly before the new hour starts

I've been reading the recent blogpost "Winding down my Debian involvement" by Michael Stapelberg. Sad details aside, it's been mentioned that within Debian infrastructure batch jobs run four times a ...
0
votes
0answers
157 views

How do I delete a single one-off file without disturbing the batch-job I'm making?

As a follow up to this question, how do I delete a single file without losing the selection that I've already got? for i in $(seq 1 5); do touch "$i.jpg"; done; mkdir subdir Let's say you marked 1....
0
votes
0answers
18 views

Batch moving a custom set of images with Ranger? [duplicate]

This is a question for the Ranger (file manager) Let's say that I have a directory with five files.. for i in $(seq 1 5); do touch "$i.jpg"; done; mkdir subdir How would I move 1.jpg, and 4.jpg, ...
0
votes
1answer
44 views

How to run the same command to execute a file in multiple directories?

I want to do the following, I have a set of directories: e.g) 400K 500K 600K and so on In each directory I have a "run.pbs" file, that I want to submit through batch with "qsub run.pbs". I was ...
-1
votes
1answer
818 views

Copy file from Windows to Linux WHEN A SHELL SCRIPT IS EXECUTED

I have a program on Linux server, which runs a shell script on Linux server. Say "/tmp/ShellScript.sh" Now I want to copy a file placed on my Windows Machine location "C:\WindowsFolder\FileToCopy" to ...
1
vote
1answer
55 views

batch rename folders with value from json value in package.json

This is a bit of a crazy one, but I am wondering if it is even possible. I have maybe 300 folders in my /var/www folder that were all renamed to the same name [Name](Num) so kindred (19) for example. ...
1
vote
1answer
326 views

batch script run command if available memory less than X mb

I would like write a batch script that checks used or available memory allow me to run commands if available memory less than X mb. I googled but page they refer didn't work for me I am using centos ...
0
votes
3answers
60 views

Selectively batch copy files

I've got a directory containing these files: RG000909-A1_1.jpg RG000909-A2_1.jpg RG000909-A3_1.jpg RG000909-A4_1.jpg RG000909-A5_1.jpg RG000909-A6_1.jpg RG000909-A7_1.jpg RG000909-A8_1.jpg ...
0
votes
1answer
189 views

How can I download and save all the images shown in a web page?

I want to download the images shown in a web page, for example https://www.wikiart.org/en/paintings-by-style/art-nouveau-modern, (the ones under the All Artworks tab), and store them in a folder (e.g. ...
4
votes
0answers
309 views

Distinguish between error and “success” in scanimage-batch

I'm running a little script with the scanimage batch-command on a remote server and would like to know if and how the scan has been done the batch. Therefore the script requires a proper "error"-...
1
vote
0answers
186 views

How to Sequentially batch sas codes in queue?

I have couple of SAS jobs that I need to run sequentially and some in parallel. I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few ...
1
vote
2answers
920 views

Count unique lines only to a set pattern

How can I count the unique log lines in a text file only until the first "-" and print the line with the count org.springframework. - initialization started org.springframework. - initialization ...
2
votes
2answers
43 views

Batch conversion via filelist.csv

I need to batch convert some video files from H264 to H265. I know my ffmpeg command line, I have a csv of file names with a relative path (I need the filelist to drive it as some files in the ...
2
votes
0answers
567 views

How to reliably detect stdin pipe in Bash in batch mode?

I need a way to test whether input to my Bash script has been piped to stdin when the script is run in batch mode. The method I use when I run the script interactively (the test [ -p /dev/stdin ]) ...
1
vote
0answers
30 views

Create a file at time 't' after PBS job execution begins

I submit a PBS job for 02:00:00 hours. I need to create a file in the PBS working directory at a specified time t (say 01:30:00 hours) after the job has begun to ensure that the job terminates ...
0
votes
1answer
71 views

Closing a terminal from an app that was started in that terminal

At work we have a supercomputer accessible via SSH. We login, reserve the cores using bsub command with parameters, and the latter creates (from doc) "an interactive bash session with pseudo-terminal....
1
vote
1answer
2k views

How to set the path of a batch environment to find the location of the “qsub” command

I have a program, called "minvappmpi", which is parallelised and itself calls qsub as a subroutine. When I use the program on the cluster, it throws the error qsub: command not found If I want to ...
1
vote
1answer
417 views

Increase allocated nodes for queued jobs on a PBS cluster

I have submitted a job on a PBS managed cluster using qsub -l nodes=5:ppn=16. My job is queued and is waiting for other jobs to be completed. Is there a command that allows me change the number of ...
1
vote
1answer
100 views

Rename command to delete substring

I am trying to batch rename the following files: art-faculty-3_29060055362_o.jpeg fine-arts-division-faculty-2016-2017-5_29165851925_o.jpeg theatre-faculty-2016-2017-1_29132529356_o.jpeg art-...
1
vote
1answer
2k views

How to run batch sas job in unix sas?

I have 5 SAS jobs that I need to run sequentially, one after the other. I typically type in nohup sas filename1.sas & in the command line to run and manually check for progress every few hours. ...
5
votes
3answers
1k views

atd, batch // Setting the load limiting factor

I am launching non interactive jobs using batch, and I would like to increase the load limiting factor in order to use all 8 of my cores. I am on Ubuntu 16.04 LTS. From what I understand, batch uses ...
0
votes
2answers
357 views

Job Priority in Batch Jobs

I do not have administrative privileges on the cluster that i am using. I'd like to prioritize some of the jobs that I have submitted. How do i incorporate that into my batch file: #!/bin/bash #$ -...
3
votes
1answer
227 views

filter document via aspell

I need to get a clean txt document and my first approach is to use aspell. The issue is I need it on batch, no interactive mode. Every txt file is piped to aspell and must be returned a new document ...
1
vote
1answer
93 views

submitting jobs to get 3 nodes running parallel executions

I have a submit script looks like below, it tries to run a large number of instances of csce.py in backgrounds with 3 nodes.... in a laptop, this usually could successfully automatically distribute ...
0
votes
1answer
477 views

How to make a batch file answer a user prompt

I have a batch file that runs a command, and one of those commands requires you to type in yes and hit enter. Is there any way that the batch file can do this?
1
vote
2answers
121 views

Batch rename long filenames of variable length with years

I have a bunch of files in a single directory that I would like to rename as follows, example existing names: 1937 - Snow White and the Seven Dwarves.avi 1940 - Pinocchio.avi Target names: Snow ...
0
votes
1answer
27 views

rename all files in a folder deleting duplicate string-parts

I have a folder with lots of subfolders, that contain two times the same but slightly different: Movie1 {Action}{Adventure}{Sci-Fi}{Thriller}{Science Fiction}/ Movie2 {Action}{Adventure}{Thriller}{...
52
votes
6answers
93k views

Why is scp so slow and how to make it faster?

I'm trying to copy a batch of files with scp but it is very slow. This is an example with 10 files: $ time scp cap_* user@host:~/dir cap_20151023T113018_704979707.png 100% 413KB 413.2KB/s 00:00 ...
2
votes
0answers
473 views

Need a script for automation the convert a lot number audio files to another format [closed]

I have a lot number audio files in MP3 the proprietary format, I want to convert them to 'opus' the free and high quality format, with keep metadata also. My selection command-line programs are SoX (...
1
vote
2answers
429 views

multiple files rename - filenames patterns [duplicate]

I have following image files in some directory which I want to be renamed: 8 -rw-rw-r-- 1 6661 sep 24 10:28 dbConnectionOkBostjan.png 8 -rw-rw-r-- 1 6548 sep 24 10:29 dbConnectionErrorBostjan.png 8 -...
2
votes
1answer
196 views

How to have a process queue in AIX like xargs with “--max-procs”?

Consider a simple processing queue like: cat list.txt | xargs -n1 -P20 process.sh (-P or --max-procs) How to have something like that in AIX ?
3
votes
0answers
7k views

qsub job in state Eqw error: can't chdir to directory: No such file or directory

I'm trying to submit a job on a cluster via qsub, but it gets stuck in state "Eqw" with error message: $ qstat -j 466 | grep error error reason 1: 09/18/2015 17:12:32 [1125:3453]: error: ...
8
votes
1answer
5k views

Is it possible to pause jobs submitted via qsub?

I am running my jobs on a small cluster. I submitted them via qsub. Now my labmates need resources more urgently than me, so I need to either kill my jobs or pause them, if possible. Is there a way ...
0
votes
1answer
832 views

How do I run a batch file at a specific time?

I have tried using the batch and at commands with no luck. I have a batch file in a location, lets say "$DD/batchfile.batch". Here is what I have tried. batch -f $DD/batchfile.batch 10:00 today at -f ...
15
votes
4answers
26k views

How to download files and folders from Onedrive using wget?

How to use wget to download files from Onedrive? (and batch files and entire folders, if possible)
1
vote
1answer
2k views

Current directory ./ in qsub?

It seems that I cannot use ./ in qsub as in qsub -q hpc-pool ./myScript.sh where myScript.sh contains several ./. After checking, ./ somewhat is translated to ~/. Why is this the case?
9
votes
2answers
13k views

How to batch resize all images in a folder (including subfolders)?

I have huge, 12GB, gallery on the server, full of images in various subfolders. Those files are too big and not used in full resolution. I need to resize all the images down to 820px wide (keeping ...
1
vote
2answers
1k views

Mac Terminal - rename *.png.png to *.png

I have a bunch of images that I'd like to rename as follows: *.png.png --> *.png *.jpeg.jpg --> *.jpg *.JPEG --> *.jpg The only thing I've tried thus far is mv *.png.png *.png, but I knew ...
0
votes
2answers
461 views

Intro to shell scripts with basic application: how do I set my computer to make an archive of a particular folder every day?

I've got a situation where I need to give some people access to my dropbox folder. Already so far they have deleted huge folders full of important data. Their computer skills could use a bit of work,...
7
votes
2answers
32k views

I don't know how to cancel job

I'm running some simulation by using the Linux cluster but I have some problem in my job. I want to cancel any job(ID : 750, 752, 753 name : gib) but I don't know how to cancel this job. Enclosed ...
7
votes
2answers
5k views

PBS equivalent of 'top' command: avoid running 'qstat' repeatedly

When I run several jobs on a head node, I like to monitor the progress using the command top. However, when I'm using PBS to run several jobs on a cluster, top will of course not show these jobs, and ...
3
votes
2answers
2k views

How can I queue processes?

I have a surveillance camera and a program records videos when motion is detected. Basically, this program saves the video in a very heavy format. My solution was to call a script that converts the ...