2

I have the following zip archive structure:

$ unzip -l Undetermined_S0_L004_R1_001_fastqc.zip 
Archive:  Undetermined_S0_L004_R1_001_fastqc.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
        0  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/
        0  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/
        0  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/
     1197  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/fastqc_icon.png
     1450  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/warning.png
     1561  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/error.png
     1715  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Icons/tick.png
      782  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/summary.txt
     9095  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_base_quality.png
    14381  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_tile_quality.png
    23205  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_sequence_quality.png
    30978  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_base_sequence_content.png
    31152  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_sequence_gc_content.png
     7861  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/per_base_n_content.png
    18356  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/sequence_length_distribution.png
    23040  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/duplication_levels.png
     9096  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/adapter_content.png
    58683  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/Images/kmer_profiles.png
   355919  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/fastqc_report.html
   301092  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/fastqc_data.txt
    10117  10-10-14 14:44   Undetermined_S0_L004_R1_001_fastqc/fastqc.fo
 --------                   -------
   899680                   21 files

How is it possible to use fastqc_data.txt with crimson in parallel, because at the moment I get the following error:

find `pwd`/*_fastqc.zip -type f | parallel -j 3 unzip -c {} {}/fastqc_data.txt | crimson fastqc {} | less

Usage: crimson fastqc [OPTIONS] INPUT [OUTPUT]

Error: Invalid value for "input": Path "{}" does not exist.
4
  • In most shells you need to quote the {}, so use e.g. \{\}.
    – dirkt
    Commented Dec 12, 2016 at 12:09
  • @dirkt Among the common and even less common shells, you only need to quote {} in fish. This is not the problem. Commented Dec 12, 2016 at 22:41
  • I tried all combination of \{\}, but still I have the same problem. What else did I miss?
    – user977828
    Commented Dec 12, 2016 at 23:41
  • WRong version of Parallel? E.g. not GNU? Try parallel --version
    – Ole Tange
    Commented Dec 13, 2016 at 3:06

1 Answer 1

2

You have a pipeline made of four commands:

  • find, which lists zip files.
  • parallel, which invokes unzip to extract one file in each zip file. Given that {} is replaced by the path to the zip file, you attempt to extract files like home/user977828/stuff/Undetermined_S0_L004_R1_001_fastqc.zip/fastqc_data.txt from the archive (if the current directory is /home/user977828/stuff).
  • crimson, which receives a jumble of the extracted files on standard input, and is invoked with the arguments fastqc and {},
  • less.

parallel only substitutes {} in its arguments. It can't do anything about the other parts of your pipeline. If you want to invoke crimson on each fastqc_data.txtfile separately, you need to pass a pipeline from unzip to crimson as an argument to parallel.

find *_fastqc.zip -type f | sed 's/\.zip$//' |
parallel -j 3 'unzip -c {}.zip {}/fastqc_data.txt | crimson fastqc /dev/stdin' |
less
4
  • I changed your command to write to an output file find pwd/test/*_fastqc.zip -type f | sed 's/\.zip$//' | parallel -j 2 'unzip -c {}.zip {}/fastqc_data.txt | crimson fastqc /dev/stdin > {}.json' but now I got caution: filename not matched: /home/lorencm/test/Undetermined_S0_L001_R1_001_fastqc/fastqc_data.txt
    – user977828
    Commented Dec 14, 2016 at 4:56
  • find pwd/test/*_fastqc.zip does not work, but find *_fastqc.zip works. Any idea why?
    – user977828
    Commented Dec 14, 2016 at 21:23
  • 1
    @user977828 Whatever parallel receives as input is used twice: once to build the name of the zip file ({}.zip) and once as the name of the directory inside the zip. If you want them to be different, you have to extract the relevant part, e.g. find … | parallel 'dir={}; dir=${dir%.zip}; dir=${dir##*/}; unzip -c {} "$dir/fastqc_data.txt" | …' Commented Dec 14, 2016 at 22:25
  • @user977828 Use parallel 'set -x; … to print a trace of what the shell launched by parallel is executing. Commented Dec 14, 2016 at 22:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .