4

I am trying to do the following:

  1. Loop through a list of files
  2. Edit each file and output a "trimmed" file
  3. Horizontally join each trimmed file in turn to a pre-made master file

I have steps 1 & 2 working, but can't get 3 to work.

Example

MASTER FILE:

Col1   Col2
A       1
B       1
C       2

TRIMMED FILE for file S1:

S1.Col3    S1.Col4
0            1
1            1
1            1

OUTPUT I WANT after S1 & S2 have been joined to MASTER:

Col1   Col2   S1.Col3   S1.Col4   S2.Col3   S2.Col4
A       1       0         1          0        1
B       1       1         1          1        0
C       2       1         1          0        0

After each loop, I've tried running something like:

paste MASTER.txt S1.txt > MASTER.txt

However, instead of the above, I get a file with only the last S file's data. I tried running this without outputting and outside of the loop, everything works fine. Thanks in advance for any answers.

For completion, here is a very simplified breakdown of my for loop.

for FILE in FILELIST
do
    cut -f4,6 $FILE > ${FILE}_trimmed.txt
    paste MASTER.txt ${FILE}_trimmed.txt > MASTER.txt
done
  • hint: the > MASTER.txt clobbers the file before paste can get to its contents. – iruvar Dec 22 '15 at 4:54
  • I see what you're saying, I think. I initially tried paste MASTER.txt ${FILE}_trimmed.txt > MASTER2.txt but that obviously doesn't work if I'm looping through files. I just end up with the final S file joined to the MASTER. So, still unsure how to get the result I'm looking for. I'm used to working in R, where I could possibly store as a temp variable and then output the file at the end. Unsure if that is plausible here or a good way of doing this... – Gaius Augustus Dec 22 '15 at 5:24
  • 1
    simplistically, change paste MASTER.txt ${FILE}_trimmed.txt > MASTER.txt to paste MASTER.txt ${FILE}_trimmed.txt > MASTER2.txt && mv MASTER2.txt MASTER.txt – iruvar Dec 22 '15 at 5:28
  • I see. That's perfect. I've created an answer, but will delete if you would prefer to explain it more thoroughly. – Gaius Augustus Dec 22 '15 at 5:39
  • No problems, +1 as well – iruvar Dec 22 '15 at 5:43
2

Thank you to 1_CR , who answered this in the comments.

The > MASTER.txt is truncating the file before the paste actually completes the horizontal join. The answer is to output to a temporary file (e.g. MASTER2.txt) and then rename it (with the mv command) to the proper file name.

My Old Code:

paste MASTER.txt ${FILE}_trimmed.txt > MASTER.txt

Fixed Code:

paste MASTER.txt ${FILE}_trimmed.txt > MASTER2.txt && mv MASTER2.txt MASTER.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.