I need to concatenate files based upon the name of the subdirectory they are in. The subdirectories sometimes have duplicates and sometimes don't. The file structure looks like this:

  1. RootDir
    • 633
      • 633_S1_L001_R1_001.fastq
      • 633_S1_L001_R2_001.fastq
    • 739
      • 739_S1_L001_R1_001.fastq
      • 739_S1_L001_R2_001.fastq
    • 739(1)
      • 739_S1_L001_R1_001.fastq
      • 739_S1_L001_R2_001.fastq
    • 739(2)
      • 739_S1_L001_R1_001.fastq
      • 739_S1_L001_R2_001.fastq
    • 834
      • 834_S3_L001_R1_001.fastq
      • 834_S3_L001_R2_001.fastq
    • 834(1)
      • 834_S7_L001_R1_001.fastq
      • 834_S7_L001_R2_001.fastq

It is the 3 number prefix and the R-number that matters, and the concat should be done on matching R#'s, with the S-number being variable.

  1. No concatenation would need to be done to files in dir 633
  2. All 3 R1 files in the 739 dirs would need to be concatenated in order (739 first, 739(1)...)
  3. The output would need to go into a subdir of the subdir (/RootDir/739/739/*)
  4. In the end, each final output dir would have an concatenated R1 file and an R2 file.

I would greatly appreciate any help in the right direction. Also note that .fastq files are simply ASCII txt files.

edit: I saw this somewhat related post, but have had no success in using the code there due to the multiple matching folders issue.

edit2: Neither of these solutions are working for me. I am slowly combining ideas both have given me and will post my eventual solution here.

3 Answers 3


according to your example, any directory that has "duplicates" has a duplicate ending in "(1)", so:

for dir in ???\(1\)/; do 
    for i in 1 2; do
        echo "mv ${base}/$f ${base}/$f.bak"
        echo "cat ${base}*/${f}* > ${base}/$f"

Remove the "echo"s when you're ready

  • Sometimes there are up to 4 duplicates. eg, dir, dir(1),dir(2),dir(3) Oct 9, 2013 at 14:55
  • This should grab them all in the cat line with ${base}* Oct 9, 2013 at 15:49

There is no use for any text processing tool other than cat, since all you're doing with file content is to concatenate them.

We start in RootDir. I assume that in NNN(K), K ≤ 999999999, and K is written with no leading 0. I construct the patterns with ([0-9]), ([0-9][0-9]), etc., so as to concatenate NNN(9) before NNN(10).

while [ ${#pattern_prefix} -le 13 ]; do
  # Iterate over the NNN(K) directory where K has a certain number of digits
  for dir in $pattern_prefix\); do
    if ! [ -d "$dir" ]; then break; fi
    # If this is the first NNN(K) directory we meet for this NNN, create the base
    if [ ! -d "$target" ]; then
      mkdir "$target"
      cp -p "$base/"*.fastq "$target"
    # For each file in NNN(K), determine the target file and append
    for f in "$dir/"*.fastq; do
      stem=${f##*/}; stem=${f#*_*_}
      set -- "$target/"*_*_"$stem"
      cat "$f" >>"$1"
  • Interesting. Given that sometimes the prefix can be sometimes also four numbers with a (caps)letter, would just changing the pattern_prefix variable to '[0-9][0-9][0-9][0-9][A-Z]...' work? I am thinking not, so maybe one script for the 3 number prefixes and another for the four number one letter prefixes? I also removed the statement/tags about awk/sed because you are correct about bash/cat being the only things really needed. Oct 9, 2013 at 14:49
  • @Carpe_Noctem You could even change it to pattern_prefix=*\([0-9] if all the file names containing an open parenthesis are of the desired form. Oct 9, 2013 at 14:54

If I'm understanding the question correctly, you want something like this:

for D in ???
  [[ -d $D?* ]] || continue
  mkdir $D/$D
  for F in $D/*
    cat $D*/$F > $D/$D/$F

That assumes no spaces in directory of file names, and that all the duplicated files will appear at least in the 3-char directory. If you might have a filename which is only in the 111(1) directories, replace

  for F in $D/*


  for F in $( find $D* -printf "%f\n" | sort -u )

so you'll get a list of unique names in 111, 111(1), etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.