Sorry if this question has been asked before. I am new to all of this.

I would like to concatenate all files from different folders that contain R1 at a specific position in their filenames. My attempts so far are not working as some file names have a different S number.

Folder 1


Folder 2

  • 2
    Welcome to the site. Please explicitly state which of the above mentioned filenames you want to include in your concatenation, as they not only differ in their "S number" but also in the number after the 952.
    – AdminBee
    Jun 30 at 13:39
  • Also, why do you want to do this? Are you sure you need it? Every tool I can think of that works on this sort of input can handle paired-end reads in separate files. In fact, I would expect the tools to break if you have the paired reads in the same file. The different S values would suggest they are different samples, so again, are you sure you want to combine them?
    – terdon
    Jun 30 at 13:51
  • Thank you. So I would like to concatenate all the files that have 56890 and R1 from different directories together. Then all the files that have 56890 and R2 from different directories together. Then all the files that have 53929 and R1 from different directories together. Then all the files that have 53929 and R2 from different directories together. So the key identifier is the number after the 952 I would like to do this before they are the same samples that have been run multiple times to increase the sequencing depth. Hope this makes sense
    – eve_hill
    Jun 30 at 14:26
  • I have been able to do this using bash script that I had help writing. The only problem is that using * assumes the rest of the file name is identical and unfortunately they are not as they differ in the letters following S
    – eve_hill
    Jun 30 at 14:33
  • You ran the samples several times to increase the sequencing depth? That seems odd, but OK. Have you ever done this before? I've never tried this but I would expect that to greatly complicate the deduplication step. Also, can your aligner deal with that? Won't it complain because the reads have different sample names?
    – terdon
    Jun 30 at 14:41

$ cat 952_53929_S*R1._001.fastq.gz >> file-name


If you only have a couple of sample names, you can do it manually:

cat folder*/952_53929_S*_R1_*.fastq.gz > 952_53929_combined_L001_R1_001.fastq.gz
cat folder*/952_53929_S*_R2_*.fastq.gz > 952_53929_combined_L001_R2_001.fastq.gz
cat folder*/952_56890_S*_R1_*.fastq.gz > 952_56890_combined_L001_R1_001.fastq.gz
cat folder*/952_56890_S*_R2_*.fastq.gz > 952_56890_combined_L001_R2_001.fastq.gz

If you have more, that isn't very practical. So you can instead collect the names yourself:

$ for f in */952_*.fastq.gz; do fname=$(basename "$f"); echo ${fname%%_S*} ; done | sort | uniq

That will give you your prefixes, and you can therefore do:

for f in */952_*.fastq.gz; do 
    fname=$(basename "$f"); 
    echo ${fname%%_S*} ; 
done | 
    sort | 
        uniq | 
            while read prefix; do 
                cat */"$prefix"*_R1_*.fastq.gz > "$prefix"_combined_L001_R1_001.fastq.gz; 
                cat */"$prefix"*_R2_*.fastq.gz > "$prefix"_combined_L001_R2_001.fastq.gz; 
  • Thank very much you! This looks like it would work and answers my question. The only problem is I have 200 samples so would I need to put all 200 into the prefix. This will take a while. I wonder if I can email you as If you see my script maybe it will make more sense. Sorry I am very new to this and therefore not amazing with terminology and phrasing my question/ problem very well.
    – eve_hill
    Jun 30 at 15:35
  • @eve_hill the second approach will work for any number of samples. If the 952 canm also change, just change the beginning to for f in */*.fastq.gz; do . By the way, you might be interested in our sister site: Bioinformatics.
    – terdon
    Jun 30 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.