2

My directory structure is given below. I need to move all the folders from Test3 to Test2 and concatenate the files with same names

[jg@hpc Test2]$ tree
.
|-- Sample_1008
|   |-- 1008_ATCACG_L002_R1_001.fastq
|   |-- 1008_ATCACG_L002_R2_001.fastq
|   |-- 1008_ATCACG_L006_R1_001.fastq
|   `-- 1008_ATCACG_L006_R2_001.fastq
`-- Sample_1009
    |-- 1009_CGATGT_L002_R1_001.fastq
    |-- 1009_CGATGT_L002_R2_001.fastq
    |-- 1009_CGATGT_L006_R1_001.fastq
    `-- 1009_CGATGT_L006_R2_001.fastq

[jg@hpc Test3]$ tree
.
|-- Sample_1008
|   |-- 1008_ATCACG_L002_R1_001.fastq
|   |-- 1008_ATCACG_L002_R2_001.fastq
|   |-- 1008_ATCACG_L006_R1_001.fastq
|   `-- 1008_ATCACG_L006_R2_001.fastq
`-- Sample_1009
    |-- 1009_CGATGT_L002_R1_001.fastq
    |-- 1009_CGATGT_L002_R2_001.fastq
    |-- 1009_CGATGT_L006_R1_001.fastq
    `-- 1009_CGATGT_L006_R2_001.fastq

I tried

mv Test3/* /auto/dr-lc_sa1/Data/Test2

nothing worked but when I tried

cp -r Test3/* Test2/

It overwrites.

I want the files to be concatenated. At the end I need to have one Test2 directory and under every sample and their fastq files in the Test3 directory will be concatenated to corresponding fastq files in Test2 directory.

1 Answer 1

3

There's no built-in way to concatenate a file and remove it, you'll have to break it into two steps.

In zsh, or in bash ≥4 after running shopt -s globstar, or in ksh after running set -o globstar:

cd Test3
for x in **/*.fastq; do
  cat "$x" >>"/auto/dr-lc_sa1/Data/Test2/$x" && rm "$x"
done

Without ** to recurse into subdirectories, use find.

cd Test3
find . -name '*.fastq' -exec sh -c 'cat "$0" >>"/auto/dr-lc_sa1/Data/Test2/$0" && rm "$0"' {} \;

If Test2 and Test3 are on the same filesystem and there are many files under Test3 that don't have a corresponding file in the destination, you can save some execution time by moving the file instead of concatenating it onto an empty file:

for x in **/*.fastq; do
  if [ -s "../Test2/$x" ]; then
    cat "$x" >>"/auto/dr-lc_sa1/Data/Test2/$x" && rm "$x"
  else
    mv "$x" "/auto/dr-lc_sa1/Data/Test2/$x"
  fi
done
8
  • I get this error [jg@hpc Data]$ sh move_and_concatenate.sh : No such file or directoryne 1: cd: Test3 'ove_and_concatenate.sh: line 2: syntax error near unexpected token do 'ove_and_concatenate.sh: line 2: for x in */.fastq; do
    – Jana
    Nov 16, 2012 at 22:43
  • I'm not bothered about removing files in Test3. File names in both Test2 and Test3 are same. All I want them to be concatenated.
    – Jana
    Nov 16, 2012 at 22:47
  • @Jana If you want to keep the data under Test3, remove the rm or mv part and keep only cat. The error message doesn't make any sense if you copy-pasted the code: what exactly did you run? Nov 16, 2012 at 22:53
  • @Giles This is what I did [jg@hpc Data]$ cat move_and_concatenate.sh cd Test3 for x in */.fastq; do cat "$x" >>"/auto/dr-lc_sa1/Data/Test2/$x" done[jg@hpc Data]$ sh move_and_concatenate.sh : No such file or directoryne 1: cd: Test3 'ove_and_concatenate.sh: line 2: syntax error near unexpected token do 'ove_and_concatenate.sh: line 2: for x in */.fastq; do
    – Jana
    Nov 16, 2012 at 22:59
  • @Jana I don't understand where those ' are coming from. What shell is sh? You should start the script with a shebang line. You'll need ksh93 or bash or zsh to use **. Make sure the script doesn't have Windows line endings. Nov 16, 2012 at 23:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .