4

I am trying to do the following:

  1. Loop through a list of files
  2. Edit each file and output a "trimmed" file
  3. Horizontally join each trimmed file in turn to a pre-made master file

I have steps 1 & 2 working, but can't get 3 to work.

Example

MASTER FILE:

Col1   Col2
A       1
B       1
C       2

TRIMMED FILE for file S1:

S1.Col3    S1.Col4
0            1
1            1
1            1

OUTPUT I WANT after S1 & S2 have been joined to MASTER:

Col1   Col2   S1.Col3   S1.Col4   S2.Col3   S2.Col4
A       1       0         1          0        1
B       1       1         1          1        0
C       2       1         1          0        0

After each loop, I've tried running something like:

paste MASTER.txt S1.txt > MASTER.txt

However, instead of the above, I get a file with only the last S file's data. I tried running this without outputting and outside of the loop, everything works fine. Thanks in advance for any answers.

For completion, here is a very simplified breakdown of my for loop.

for FILE in FILELIST
do
    cut -f4,6 $FILE > ${FILE}_trimmed.txt
    paste MASTER.txt ${FILE}_trimmed.txt > MASTER.txt
done
6
  • hint: the > MASTER.txt clobbers the file before paste can get to its contents.
    – iruvar
    Commented Dec 22, 2015 at 4:54
  • I see what you're saying, I think. I initially tried paste MASTER.txt ${FILE}_trimmed.txt > MASTER2.txt but that obviously doesn't work if I'm looping through files. I just end up with the final S file joined to the MASTER. So, still unsure how to get the result I'm looking for. I'm used to working in R, where I could possibly store as a temp variable and then output the file at the end. Unsure if that is plausible here or a good way of doing this... Commented Dec 22, 2015 at 5:24
  • 1
    simplistically, change paste MASTER.txt ${FILE}_trimmed.txt > MASTER.txt to paste MASTER.txt ${FILE}_trimmed.txt > MASTER2.txt && mv MASTER2.txt MASTER.txt
    – iruvar
    Commented Dec 22, 2015 at 5:28
  • I see. That's perfect. I've created an answer, but will delete if you would prefer to explain it more thoroughly. Commented Dec 22, 2015 at 5:39
  • No problems, +1 as well
    – iruvar
    Commented Dec 22, 2015 at 5:43

1 Answer 1

2

Thank you to 1_CR , who answered this in the comments.

The > MASTER.txt is truncating the file before the paste actually completes the horizontal join. The answer is to output to a temporary file (e.g. MASTER2.txt) and then rename it (with the mv command) to the proper file name.

My Old Code:

paste MASTER.txt ${FILE}_trimmed.txt > MASTER.txt

Fixed Code:

paste MASTER.txt ${FILE}_trimmed.txt > MASTER2.txt && mv MASTER2.txt MASTER.txt

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .