1

Disclaimer: This is a more general question of the one I asked on biostars.org about parallel and writing to file.

When I run a program (obisplit from obitools package) sequentially, it reads one file and creates a number of files based on some criterion (not important here) in the original file:

input_file.fastq
     |____ output_01.fastq
     |____ output_02.fastq
     |____ output_03.fastq

However, when I split the input file and run them in parallel (version from ubuntu repo: 20141022),

find . * | grep -P "^input_file" | parallel -j+3 obisplit -p output_{/.}_ -t variable_to_split_on {/}

I would expect to get files

input_file_a.fastq
     |____ output_input_file_a_01.fastq
     |____ output_input_file_a_02.fastq
     |____ output_input_file_a_03.fastq
input_file_b.fastq
     |____ output_input_file_b_01.fastq
     |____ output_input_file_b_02.fastq
     |____ output_input_file_b_03.fastq
input_file_c.fastq
     |____ output_input_file_c_01.fastq
     |____ output_input_file_c_02.fastq
     |____ output_input_file_c_03.fastq

but the output is only printed to console.

Is there something inherent in parallel which causes this printing to console or could this be the way obisplit is behaving for whatever reason? Is there a way to convince each core commandeered by parallel to print to a specific file instead of the console?

  • 1
    Does parallel --dry-run output what you expect to be run? – Ole Tange May 25 '17 at 22:27
1

It sound as if obisplit behaves differently if output is redirected.

You can ask GNU Parallel to output to files:

seq 10 | parallel --results output_{} echo this is input {} >/dev/null

(or if your version is older:

seq 10 | parallel echo this is input {} '>' output_{}

)

It will create output_#,output_#.err,output_#.seq.

  • Sorry to have wasted your time, but your comment under my question nudged me into the right direction. I did a dry run and actually ran one part of the output and it was outputting into the console. This got me thinking and upon closer inspection, I was calling the wrong .fastq file :facepalm:. When the right file was called, things folded into place nicely. Thank you for this wonderful tool, it's saving me temporal tonnes of time. – Roman Luštrik May 26 '17 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.