Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

My directory structure is given below. I need to move all the folders from Test3 to Test2 and concatenate the files with same names

[jg@hpc Test2]$ tree
.
|-- Sample_1008
|   |-- 1008_ATCACG_L002_R1_001.fastq
|   |-- 1008_ATCACG_L002_R2_001.fastq
|   |-- 1008_ATCACG_L006_R1_001.fastq
|   `-- 1008_ATCACG_L006_R2_001.fastq
`-- Sample_1009
    |-- 1009_CGATGT_L002_R1_001.fastq
    |-- 1009_CGATGT_L002_R2_001.fastq
    |-- 1009_CGATGT_L006_R1_001.fastq
    `-- 1009_CGATGT_L006_R2_001.fastq

[jg@hpc Test3]$ tree
.
|-- Sample_1008
|   |-- 1008_ATCACG_L002_R1_001.fastq
|   |-- 1008_ATCACG_L002_R2_001.fastq
|   |-- 1008_ATCACG_L006_R1_001.fastq
|   `-- 1008_ATCACG_L006_R2_001.fastq
`-- Sample_1009
    |-- 1009_CGATGT_L002_R1_001.fastq
    |-- 1009_CGATGT_L002_R2_001.fastq
    |-- 1009_CGATGT_L006_R1_001.fastq
    `-- 1009_CGATGT_L006_R2_001.fastq

I tried

mv Test3/* /auto/dr-lc_sa1/Data/Test2

nothing worked but when I tried

cp -r Test3/* Test2/

It overwrites.

I want the files to be concatenated. At the end I need to have one Test2 directory and under every sample and their fastq files in the Test3 directory will be concatenated to corresponding fastq files in Test2 directory.

share|improve this question
add comment

1 Answer 1

up vote 3 down vote accepted

There's no built-in way to concatenate a file and remove it, you'll have to break it into two steps.

In zsh, or in bash ≥4 after running shopt -s globstar, or in ksh after running set -o globstar:

cd Test3
for x in **/*.fastq; do
  cat "$x" >>"/auto/dr-lc_sa1/Data/Test2/$x" && rm "$x"
done

Without ** to recurse into subdirectories, use find.

cd Test3
find . -name '*.fastq' -exec sh -c 'cat "$0" >>"/auto/dr-lc_sa1/Data/Test2/$0" && rm "$0"' {} \;

If Test2 and Test3 are on the same filesystem and there are many files under Test3 that don't have a corresponding file in the destination, you can save some execution time by moving the file instead of concatenating it onto an empty file:

for x in **/*.fastq; do
  if [ -s "../Test2/$x" ]; then
    cat "$x" >>"/auto/dr-lc_sa1/Data/Test2/$x" && rm "$x"
  else
    mv "$x" "/auto/dr-lc_sa1/Data/Test2/$x"
  fi
done
share|improve this answer
    
I get this error [jg@hpc Data]$ sh move_and_concatenate.sh : No such file or directoryne 1: cd: Test3 'ove_and_concatenate.sh: line 2: syntax error near unexpected token do 'ove_and_concatenate.sh: line 2: for x in */.fastq; do –  Jana Nov 16 '12 at 22:43
    
I'm not bothered about removing files in Test3. File names in both Test2 and Test3 are same. All I want them to be concatenated. –  Jana Nov 16 '12 at 22:47
    
@Jana If you want to keep the data under Test3, remove the rm or mv part and keep only cat. The error message doesn't make any sense if you copy-pasted the code: what exactly did you run? –  Gilles Nov 16 '12 at 22:53
    
@Giles This is what I did [jg@hpc Data]$ cat move_and_concatenate.sh cd Test3 for x in */.fastq; do cat "$x" >>"/auto/dr-lc_sa1/Data/Test2/$x" done[jg@hpc Data]$ sh move_and_concatenate.sh : No such file or directoryne 1: cd: Test3 'ove_and_concatenate.sh: line 2: syntax error near unexpected token do 'ove_and_concatenate.sh: line 2: for x in */.fastq; do –  Jana Nov 16 '12 at 22:59
    
@Jana I don't understand where those ' are coming from. What shell is sh? You should start the script with a shebang line. You'll need ksh93 or bash or zsh to use **. Make sure the script doesn't have Windows line endings. –  Gilles Nov 16 '12 at 23:07
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.