2

I have the following zip archive structure:

$ unzip -l Undetermined_S0_L004_R1_001_fastqc.zip 
Archive:  Undetermined_S0_L004_R1_001_fastqc.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
        0  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/
        0  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/
        0  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/
     1197  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/fastqc_icon.png
     1450  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/warning.png
     1561  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/error.png
     1715  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/tick.png
      782  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/summary.txt
     9095  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_base_quality.png
    14381  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_tile_quality.png
    23205  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_sequence_quality.png
    30978  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_base_sequence_content.png
    31152  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_sequence_gc_content.png
     7861  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_base_n_content.png
    18356  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/sequence_length_distribution.png
    23040  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/duplication_levels.png
     9096  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/adapter_content.png
    58683  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/kmer_profiles.png
   355919  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/fastqc_report.html
   301092  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/fastqc_data.txt
    10117  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/fastqc.fo
 --------                   -------
   899680                   21 files

How is it possible to use fastqc_data.txt with crimson in parallel, because at the moment I get the following error:

find `pwd`/*_fastqc.zip -type f | parallel -j 3 unzip -c {} {}/fastqc_data.txt | crimson fastqc {} | less

Usage: crimson fastqc [OPTIONS] INPUT [OUTPUT]

Error: Invalid value for "input": Path "{}" does not exist.
  • In most shells you need to quote the {}, so use e.g. \{\}. – dirkt Dec 12 '16 at 12:09
  • @dirkt Among the common and even less common shells, you only need to quote {} in fish. This is not the problem. – Gilles Dec 12 '16 at 22:41
  • I tried all combination of \{\}, but still I have the same problem. What else did I miss? – user977828 Dec 12 '16 at 23:41
  • WRong version of Parallel? E.g. not GNU? Try parallel --version – Ole Tange Dec 13 '16 at 3:06
3

You have a pipeline made of four commands:

  • find, which lists zip files.
  • parallel, which invokes unzip to extract one file in each zip file. Given that {} is replaced by the path to the zip file, you attempt to extract files like home/user977828/stuff/Undetermined_S0_L004_R1_001_fastqc.zip/fastqc_data.txt from the archive (if the current directory is /home/user977828/stuff).
  • crimson, which receives a jumble of the extracted files on standard input, and is invoked with the arguments fastqc and {},
  • less.

parallel only substitutes {} in its arguments. It can't do anything about the other parts of your pipeline. If you want to invoke crimson on each fastqc_data.txtfile separately, you need to pass a pipeline from unzip to crimson as an argument to parallel.

find *_fastqc.zip -type f | sed 's/\.zip$//' |
parallel -j 3 'unzip -c {}.zip {}/fastqc_data.txt | crimson fastqc /dev/stdin' |
less
  • I changed your command to write to an output file find pwd/test/*_fastqc.zip -type f | sed 's/\.zip$//' | parallel -j 2 'unzip -c {}.zip {}/fastqc_data.txt | crimson fastqc /dev/stdin > {}.json' but now I got caution: filename not matched: /home/lorencm/test/Undetermined_S0_L001_R1_001_fastqc/fastqc_data.txt – user977828 Dec 14 '16 at 4:56
  • find pwd/test/*_fastqc.zip does not work, but find *_fastqc.zip works. Any idea why? – user977828 Dec 14 '16 at 21:23
  • 1
    @user977828 Whatever parallel receives as input is used twice: once to build the name of the zip file ({}.zip) and once as the name of the directory inside the zip. If you want them to be different, you have to extract the relevant part, e.g. find … | parallel 'dir={}; dir=${dir%.zip}; dir=${dir##*/}; unzip -c {} "$dir/fastqc_data.txt" | …' – Gilles Dec 14 '16 at 22:25
  • @user977828 Use parallel 'set -x; … to print a trace of what the shell launched by parallel is executing. – Gilles Dec 14 '16 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.