How to move files with same name and concatenate

Question

My directory structure is given below. I need to move all the folders from Test3 to Test2 and concatenate the files with same names

[jg@hpc Test2]$ tree
.
|-- Sample_1008
|   |-- 1008_ATCACG_L002_R1_001.fastq
|   |-- 1008_ATCACG_L002_R2_001.fastq
|   |-- 1008_ATCACG_L006_R1_001.fastq
|   `-- 1008_ATCACG_L006_R2_001.fastq
`-- Sample_1009
    |-- 1009_CGATGT_L002_R1_001.fastq
    |-- 1009_CGATGT_L002_R2_001.fastq
    |-- 1009_CGATGT_L006_R1_001.fastq
    `-- 1009_CGATGT_L006_R2_001.fastq

[jg@hpc Test3]$ tree
.
|-- Sample_1008
|   |-- 1008_ATCACG_L002_R1_001.fastq
|   |-- 1008_ATCACG_L002_R2_001.fastq
|   |-- 1008_ATCACG_L006_R1_001.fastq
|   `-- 1008_ATCACG_L006_R2_001.fastq
`-- Sample_1009
    |-- 1009_CGATGT_L002_R1_001.fastq
    |-- 1009_CGATGT_L002_R2_001.fastq
    |-- 1009_CGATGT_L006_R1_001.fastq
    `-- 1009_CGATGT_L006_R2_001.fastq

I tried

mv Test3/* /auto/dr-lc_sa1/Data/Test2

nothing worked but when I tried

cp -r Test3/* Test2/

It overwrites.

I want the files to be concatenated. At the end I need to have one Test2 directory and under every sample and their fastq files in the Test3 directory will be concatenated to corresponding fastq files in Test2 directory.

Answer 1

There's no built-in way to concatenate a file and remove it, you'll have to break it into two steps.

In zsh, or in bash ≥4 after running shopt -s globstar, or in ksh after running set -o globstar:

cd Test3
for x in **/*.fastq; do
  cat "$x" >>"/auto/dr-lc_sa1/Data/Test2/$x" && rm "$x"
done

Without ** to recurse into subdirectories, use find.

cd Test3
find . -name '*.fastq' -exec sh -c 'cat "$0" >>"/auto/dr-lc_sa1/Data/Test2/$0" && rm "$0"' {} \;

If Test2 and Test3 are on the same filesystem and there are many files under Test3 that don't have a corresponding file in the destination, you can save some execution time by moving the file instead of concatenating it onto an empty file:

for x in **/*.fastq; do
  if [ -s "../Test2/$x" ]; then
    cat "$x" >>"/auto/dr-lc_sa1/Data/Test2/$x" && rm "$x"
  else
    mv "$x" "/auto/dr-lc_sa1/Data/Test2/$x"
  fi
done