I need to concatenate files based upon the name of the subdirectory they are in. The subdirectories sometimes have duplicates and sometimes don't. The file structure looks like this:

  1. RootDir
    • 633
      • 633_S1_L001_R1_001.fastq
      • 633_S1_L001_R2_001.fastq
    • 739
      • 739_S1_L001_R1_001.fastq
      • 739_S1_L001_R2_001.fastq
    • 739(1)
      • 739_S1_L001_R1_001.fastq
      • 739_S1_L001_R2_001.fastq
    • 739(2)
      • 739_S1_L001_R1_001.fastq
      • 739_S1_L001_R2_001.fastq
    • 834
      • 834_S3_L001_R1_001.fastq
      • 834_S3_L001_R2_001.fastq
    • 834(1)
      • 834_S7_L001_R1_001.fastq
      • 834_S7_L001_R2_001.fastq

It is the 3 number prefix and the R-number that matters, and the concat should be done on matching R#'s, with the S-number being variable.

  1. No concatenation would need to be done to files in dir 633
  2. All 3 R1 files in the 739 dirs would need to be concatenated in order (739 first, 739(1)...)
  3. The output would need to go into a subdir of the subdir (/RootDir/739/739/*)
  4. In the end, each final output dir would have an concatenated R1 file and an R2 file.

I would greatly appreciate any help in the right direction. Also note that .fastq files are simply ASCII txt files.

edit: I saw this somewhat related post, but have had no success in using the code there due to the multiple matching folders issue.

edit2: Neither of these solutions are working for me. I am slowly combining ideas both have given me and will post my eventual solution here.

3 Answers 3


according to your example, any directory that has "duplicates" has a duplicate ending in "(1)", so:

for dir in ???\(1\)/; do 
    for i in 1 2; do
        echo "mv ${base}/$f ${base}/$f.bak"
        echo "cat ${base}*/${f}* > ${base}/$f"

Remove the "echo"s when you're ready

  • Sometimes there are up to 4 duplicates. eg, dir, dir(1),dir(2),dir(3) Oct 9, 2013 at 14:55
  • This should grab them all in the cat line with ${base}* Oct 9, 2013 at 15:49

There is no use for any text processing tool other than cat, since all you're doing with file content is to concatenate them.

We start in RootDir. I assume that in NNN(K), K ≤ 999999999, and K is written with no leading 0. I construct the patterns with ([0-9]), ([0-9][0-9]), etc., so as to concatenate NNN(9) before NNN(10).

while [ ${#pattern_prefix} -le 13 ]; do
  # Iterate over the NNN(K) directory where K has a certain number of digits
  for dir in $pattern_prefix\); do
    if ! [ -d "$dir" ]; then break; fi
    # If this is the first NNN(K) directory we meet for this NNN, create the base
    if [ ! -d "$target" ]; then
      mkdir "$target"
      cp -p "$base/"*.fastq "$target"
    # For each file in NNN(K), determine the target file and append
    for f in "$dir/"*.fastq; do
      stem=${f##*/}; stem=${f#*_*_}
      set -- "$target/"*_*_"$stem"
      cat "$f" >>"$1"
  • Interesting. Given that sometimes the prefix can be sometimes also four numbers with a (caps)letter, would just changing the pattern_prefix variable to '[0-9][0-9][0-9][0-9][A-Z]...' work? I am thinking not, so maybe one script for the 3 number prefixes and another for the four number one letter prefixes? I also removed the statement/tags about awk/sed because you are correct about bash/cat being the only things really needed. Oct 9, 2013 at 14:49
  • @Carpe_Noctem You could even change it to pattern_prefix=*\([0-9] if all the file names containing an open parenthesis are of the desired form. Oct 9, 2013 at 14:54

If I'm understanding the question correctly, you want something like this:

for D in ???
  [[ -d $D?* ]] || continue
  mkdir $D/$D
  for F in $D/*
    cat $D*/$F > $D/$D/$F

That assumes no spaces in directory of file names, and that all the duplicated files will appear at least in the 3-char directory. If you might have a filename which is only in the 111(1) directories, replace

  for F in $D/*


  for F in $( find $D* -printf "%f\n" | sort -u )

so you'll get a list of unique names in 111, 111(1), etc.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .