Shell script that execute a program running multiple input files

Question

I am new to shell scripting thus I am looking for help.

I have a program that takes two arguments input file-name, and output file-name.

I have multiple data files with different names. I want to write a shell script that executes the program for all the data files, so every file will be treated separately.

For every output file the name should be the name of the input file concatenated with -s to indicate that is a solution file. Except that the -s should append the base part of the name, such that file.txt becomes file-s.txt.

Edit: Thanks to all the responders, the question is similar to that here The difference is that I am not on MS VS or neither have a problem with the C++ code, and I want to process the all data (Thousands of .txt files) at the same time (concurrently) on a workstation

Answer 1

You could use GNU Parallel program for that:

find . -name \*.txt | grep -v -e '\-s.txt$' | parallel echo program "{}" {.}-s.txt
program ./1 2.txt ./1 2-s.txt
program ./aaa.txt ./aaa-s.txt
program ./bbb.txt ./bbb-s.txt
program ./demo.txt ./demo-s.txt
program ./original_names.txt ./original_names-s.txt

Answer 2

It seems you want something like this:

for src; do
  dst="${src%.txt}-s.txt"
  program "$src" "$dst"
done

Call the script with the names of the files you want to process. The key part is "${src%.txt}-s.txt", it removes the .txt extension and adds -s.txt.

Answer 3

Assuming you currently invoke your program with two arguments as cmd input-file output-file, you could simply write your wrapper script as:

#!/bin/sh
for file; do 
    output="${file%.txt}"-s.txt
    cmd "$file" "$output" &
done

Then call the script with your multiple inputs as arguments. This solution assumes your input files all have .txt suffix (well, it doesn't really assume that, but the output file names will be wonky if they don't). Note that this will potentially overwrite existing output files; add some logic to check for prior existence if you're worried about that. Probably safer to create a directory and put all your output files in a fresh directory to avoid that problem. Or, if you want to read the filenames from input: (eg, you can call the script with < my-file-full-of-input-files-one-per-line ./my-script), write it as:

#!/bin/sh
while read file; do
    output="${file%.txt}"-s.txt
    cmd "$file" "$output" &
done