Concatenate files in multiple matching subdirectories

Question

I need to concatenate files based upon the name of the subdirectory they are in. The subdirectories sometimes have duplicates and sometimes don't. The file structure looks like this:

1. RootDir
   • 633
     • 633_S1_L001_R1_001.fastq
     • 633_S1_L001_R2_001.fastq
   • 739
     • 739_S1_L001_R1_001.fastq
     • 739_S1_L001_R2_001.fastq
   • 739(1)
     • 739_S1_L001_R1_001.fastq
     • 739_S1_L001_R2_001.fastq
   • 739(2)
     • 739_S1_L001_R1_001.fastq
     • 739_S1_L001_R2_001.fastq
   • 834
     • 834_S3_L001_R1_001.fastq
     • 834_S3_L001_R2_001.fastq
   • 834(1)
     • 834_S7_L001_R1_001.fastq
     • 834_S7_L001_R2_001.fastq

It is the 3 number prefix and the R-number that matters, and the concat should be done on matching R#'s, with the S-number being variable.

1. No concatenation would need to be done to files in dir 633
2. All 3 R1 files in the 739 dirs would need to be concatenated in order (739 first, 739(1)...)
3. The output would need to go into a subdir of the subdir (/RootDir/739/739/*)
4. In the end, each final output dir would have an concatenated R1 file and an R2 file.

I would greatly appreciate any help in the right direction. Also note that .fastq files are simply ASCII txt files.

edit: I saw this somewhat related post, but have had no success in using the code there due to the multiple matching folders issue.

edit2: Neither of these solutions are working for me. I am slowly combining ideas both have given me and will post my eventual solution here.

Answer 1

according to your example, any directory that has "duplicates" has a duplicate ending in "(1)", so:

for dir in ???\(1\)/; do 
    base=${dir%(*}
    for i in 1 2; do
        f=${base}_S1_L001_R${i}_001.fastq
        echo "mv ${base}/$f ${base}/$f.bak"
        echo "cat ${base}*/${f}* > ${base}/$f"
    done
done

Remove the "echo"s when you're ready

Answer 2

There is no use for any text processing tool other than cat, since all you're doing with file content is to concatenate them.

We start in RootDir. I assume that in NNN(K), K ≤ 999999999, and K is written with no leading 0. I construct the patterns with ([0-9]), ([0-9][0-9]), etc., so as to concatenate NNN(9) before NNN(10).

pattern_prefix='[0-9][0-9][0-9]([1-9]'
while [ ${#pattern_prefix} -le 13 ]; do
  # Iterate over the NNN(K) directory where K has a certain number of digits
  for dir in $pattern_prefix\); do
    if ! [ -d "$dir" ]; then break; fi
    base=${dir%\(*}
    target=$base/$base
    # If this is the first NNN(K) directory we meet for this NNN, create the base
    if [ ! -d "$target" ]; then
      mkdir "$target"
      cp -p "$base/"*.fastq "$target"
    fi
    # For each file in NNN(K), determine the target file and append
    for f in "$dir/"*.fastq; do
      stem=${f##*/}; stem=${f#*_*_}
      set -- "$target/"*_*_"$stem"
      cat "$f" >>"$1"
    done
  done
  pattern_prefix=$pattern_prefix'[0-9]'
done

Answer 3

If I'm understanding the question correctly, you want something like this:

for D in ???
do
  [[ -d $D?* ]] || continue
  mkdir $D/$D
  for F in $D/*
  do
    cat $D*/$F > $D/$D/$F
  done
done

That assumes no spaces in directory of file names, and that all the duplicated files will appear at least in the 3-char directory. If you might have a filename which is only in the 111(1) directories, replace

  for F in $D/*

with

  for F in $( find $D* -printf "%f\n" | sort -u )

so you'll get a list of unique names in 111, 111(1), etc.