I am running a command line software on multiple folder/samples.Each folder has such files *fastq.gz.

Below is an example of a folder.

Sample_EC_only/EC_only_S1_L005_I1_001.fastq.gz Sample_EC_only/EC_only_S1_L005_R1_001.fastq.gz Sample_EC_only/EC_only_S1_L005_R2_001.fastq.gz Sample_EC_only/EC_only_S1_L006_I1_001.fastq.gz Sample_EC_only/EC_only_S1_L006_R1_001.fastq.gz

I am trying to run this using gnu parallel for multiple softwares,but having issues with extracting the "ID" of the folder.

parallel -j $NSLOTS --xapply \
" echo {1}  \
/home/rob2056/software/cellranger-2.2.0/cellranger count --id  = "{basename} {1}"  \
                   --transcriptome=$ref_data \
                   --fastqs={1} \
 " ::: $TMPDIR/FASTQ/Sample* 

I want to extract for e.g. "Sample_EC_only" as a pattern from the folder inside gnu parallel. --fastqs is able to get the path using {1} ,but having issues with --id option.I have tried various options to extract a pattern from the paths in {1} but not working.

The --id parameter needs a pattern extracted from the path in {1} so that it can create a output dir.

Each {1} consists of e.g. (shown below only for one sample)


  • While GNU Parallel will work on multiline commands, it is recommended to define a bash function and call that instead. gnu.org/software/parallel/… In your case the " around {basename} {1} will be confusing to the average reader, as they are not matching. Instead the first is ending the " from the line before and the second is ended with the " before ::: – Ole Tange Sep 11 '18 at 20:53

If I understand you correctly, all you are looking for is {1/} instead of {1}. It is the "basename" of the argument. See man parallel_tutorial and the discussion of --rpl where we have that replacement strings are implemented as

     --rpl '{/} s:.*/::'

and The positional replacement strings can also be modified using / etc. So {1/} is like removing all characters upto the final /.

You can create your own replacement shorthand strings using --rpl followed by a string which begins with a tag ({/} in the example above), then a perl expression, such as the substitute command above (s:pattern:replacement:).

I'm not sure what is allowed as tags, but we can use the tutorial example {..} for a positional tag, i.e. that can be used with {number}. The perl expression to remove everything upto the last / followed by the word "Sample_" woudl be: s:.*/Sample_:: so you need to add before --xapply the arguments

--rpl '{..} s:.*/Sample_::'

and then use --id={1..} to apply this replacement to arg 1. If, for example, you want to remove the word upto the first underline _, rather than a fixed word Sample, you can use a pattern such as

--rpl '{..} s:.*/[^_]*_::'

The final command should look something like this:

parallel -j $NSLOTS --rpl '{..} s:.*/Sample_::' --xapply \
" echo {1}  \
/home/rob2056/software/cellranger-2.2.0/cellranger count --id={1/} \
                   --id2={1..} \
           --transcriptome=$ref_data \
           --fastqs={1} \
 " ::: $TMPDIR/FASTQ/Sample* 
| improve this answer | |
  • Also what would be the easiest way to extract "EC_only" from the pattern ? – Ron Sep 11 '18 at 22:45
  • See my updated answer. – meuh Sep 12 '18 at 7:54
  • I want to add another parameter inside my command for e.g. id2 which takes "Ec_only" pattern from {1} (folder name in my case). I tried the method you suggested however not sure if it works --id2= {/} s:.*/Sample_:: – Ron Sep 13 '18 at 22:43
  • I added what I think you want as a final command at the end of my answer above. You need to add a --rpl which specifies the tag and the substitution command, then use the tag later on in the command. – meuh Sep 14 '18 at 15:36
  • The tag does not work --id2={1..} .The command wont run.(included the --rpl '{..} s:.*/Sample_::' in parallel) – Ron Sep 14 '18 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.