2

I want to run a script on all the files in a folder like this:

sh script.sh *.fasta > output

however I want the outputs to be in individual files for every input. So instead of getting 5 files through the command and putting them all into a file called "output" is there a way to have the outputs be the name of the file with a suffix at the end like "_output"?

So if I had 3 files like:

file1.fasta
file2.fasta
file3.fasta

Is there a way I could run a command like this:

sh script.sh *.fasta

and output the new files like this for each input:

file1_new.fasta
file2_new.fasta
file3_new.fasta

I tried the command:

sh script.sh *.fasta > *_output.fasta

But it didn't work, I also tried assigning all the files of interest to a variable like this, with he idea that maybe I could somehow rename the outputs if I first assigned them to a variable:

allthefiles= *.fasta

But that didn't work either. I don't know what to call this question, so sorry if it's been asked already! How do I run several files through a command and have a new output for each input?

1

Normal shell, within a script or function:

for file in "$@" ; do
  if true; do 
     # do some work which reads from stdin
     # and outputs to stdout
  fi > "${file}_output.fasta" < "$file"
done

Or the more conventional (but possibly tedious)

for file in "$@" ; do
  output="${file}_output.fasta"
  some_program $file > $output
  another_program_appends $file >> $output
done

You can also do something like this with awk:

$ awk '{ print substr($0,1,20) >> FILENAME "_output.fasta" }' *fasta

The awk script outputs the first 20 characters of each line of each input file, saving the output as you expected.

  • 1
    Just use { ... } in place of the if statement. – chepner Mar 24 '16 at 0:23
  • I know. I was being demonstrative. The { ... } is more concise but doesn't explain. – Otheus Mar 24 '16 at 11:41
  • What does the if explain? You're introducing a conditional with no actual condition. Its only purpose is to group commands in order to share a file for standard output, and that's exactly what braces are for. – chepner Mar 24 '16 at 11:53
  • To demonstrate that there is nothing special about { ... } and that any shell control block can be used to redirect input/output for that block. I am 98% certain that my explanation is superior to yours by 0.000001%. – Otheus Mar 24 '16 at 12:41
1

Here's a one line solution

find ./ -name "*.fasta" -exec sh -c 'script.sh ${0} > ${0}.log' {} \;

In short, it finds the files you want, and execs your script on them. The point of using sh -c is so that the redirect character > is not interpreted directly.

  • 1
    @Otheus, thanks for the tip with multiple {}, I've had issues with it before but it seems to work. However this still needs sh -c because otherwise the shell will pick up the redirection > and try to output to {}.log – Thomas Zwaagstra Mar 23 '16 at 23:44
2
for f in *.fasta; do 
    sh script.sh "$f" > "${f%.*}_output.fasta"; 
done

# ${f%.*} strips a shortest match of `.*` from the end of "$f"
# (= strips .fasta)
2

Use make!

Write a file called GNUmakefile with the content below, but where I put the character ↦, put a tab instead (it has to be a tab, it can't be spaces).

all: $(filter-out %_output.fasta,$(wildcard *.fasta))

%_output.fasta: %.fasta
↦./script.sh $< >$@.tmp
↦mv $@.tmp $@

Now, to regenerate all the files, type make. As a bonus, if an input file hasn't changed since the output file was last generated, script.sh won't run again. If you're actively modifying script.sh and you want to regenerate the output files, add script.sh after %.fasta, this way the files will also be regenerated if the script has changed.

Explanations:

  • The first line specifies what to do when you run make all. Since it's the first line (the first target, in makefile terminology), running make with no argument does the same thing.
  • The part after all: generates the list of names of .fasta files in the current directory, and removes the ones called _output.fasta.
  • The line %_output.fasta: %.fasta starts a rule that explains how to generate a file whose name ends with _output.fasta (the target) from the corresponding .fasta file (a dependency).
  • The following tab-indented lines are the commands to run to generate the files.
  • The first line transforms the first dependency ($<) into a .tmp file.
  • The second line renames the .tmp file into the target file ($@). The reason for this two-step process is that if the generation is interrupted for any reason, this won't leave an invalid target file behind.

Note: I assume that you're using Linux. If not, you may need to install GNU make and run that instead of your system's default make command, if you want to use the code above.

If you used a different extension for the output files, it would make things a little easier.

all: $(patsubst %.fasta,%.out,$(wildcard *.fasta))

.SUFFIX: .out .fasta
.fast.out:
↦./script.sh $< >$@.tmp
↦mv $@.tmp $@

If you replace the first line by the explicit list of files (all: foo.out bar.out) then the file can be called Makefile and will run with any implementation of make.

1

You can do them in parallel with ... gnu parallel:

parallel "sh script.sh {} > {}.out" ::: *
  • Can you explain the ::: part? – Otheus Mar 24 '16 at 11:42
  • It's not a unix thing, it's just how GNU parallel says "ok, here are your args". You can also pipe them in, I believe. More info: gnu.org/software/parallel – r_2 Mar 25 '16 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.