Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I was trying to prepare data files from one large file by extracting columns and pasting into a new file.

The number of columns of the given data is very large. I want to prepare a number of data files by splitting the data.
I was trying the script below, but it is not working. I expect the error is due to specifying ranges of columns to copy.

#!/bin/bash

paste <(awk '{print $1,$2,$3,...$19,$20}' Precipitation.csv ) > aaaaa1
paste <(awk '{print $21,$22,$23,...$39,$40}' Precipitation.csv ) > aaaaa2 
paste <(awk '{print $41,$42,$43,...$99,$100}' Precipitation.csv ) > aaaaa3 

Would you please help me in correcting the code?

share|improve this question
    
You don't show what the error message is or say exactly how it's not working. Are you including the ellipses or did you just do that in order to shorten the lines for posting the question? I don't think paste with one argument usually is very useful. –  Dennis Williamson Aug 5 at 15:02

5 Answers 5

This looks like a case where cut will do better than awk:

cut -d , -f1-20 Precipitation.csv > aaa1
cut -d , -f21-40 Precipitation.csv > aaa2
cut -d , -f41-100 Precipitation.csv > aaa3

-d , specifies the delimiter (a comma, since the input is called CSV, but you can change that). -f N-M picks out fields N through M inclusive to be in the output. If there are embedded commas within any field it will break - in that case you probably want a real CSV parser, although it is possible to hack something together if that really isn't possible.

The output will use the same delimiter as the input. GNU cut supports a --output-delimiter=STRING option to set a different delimiter, but I don't think any others do. FreeBSD's cut includes a -w split-by-whitespace option that is closer to what awk does by default. If you want one of those behaviours elsewhere, transforming the delimiter before/after is probably your best bet.

share|improve this answer
    
@Micheal Homer, thank you very much. It worked! –  Abraham Aug 5 at 7:59

A follow-up on Michael Homer's answer:

To avoid reading the file several times with shells supporting process substitution (ksh, zsh, bash):

 tee < Precipitation.csv >(cut -d, -f1-20 > aaa1) >(cut -d, -f21-40 > aaa2) | 
   cut -d, -f41- > aaa3
share|improve this answer

With perl:

perl -F',' -anle 'print join ",", @F[0..19]' Precipitation.csv > aaa1
perl -F',' -anle 'print join ",", @F[20..39]' Precipitation.csv > aaa2
perl -F',' -anle 'print join ",", @F[40..99]' Precipitation.csv > aaa3
share|improve this answer

As I understand your problem, this should be all you need - and it's all processed in a single stream, so you should handle the big file much more efficiently that way.

I did a small test on this like:

{   sed -e '/\n\n/{s///w aaa3
    d' -e '};s/,/\n/20;P;G;D' |
    sed -ne 'w aaa1' -e 'n;p'
} <<DATA > aaa2
$(
    seq -s, 100 | tee - - - -
)
DATA

Then I printed the results like:

for f in 1 2 3
do  printf '\n%s\n' aaa$f
    cat aaa$f
done

And success!

aaa1
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20

aaa2
21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40
21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40
21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40
21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40
21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40

aaa3
41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100
41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100
41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100
41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100
41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100
share|improve this answer

This provides better performance as we are reading the input file only once

awk '{out1=$1; for(i=2; i<=20; i++) {out1 = (out1 FS $i)}
  out2=$21; for(i=22; i<=40; i++) {out2=(out2 FS $i)}
  out3=$41; for(i=42; i<=100; i++) {out3=(out3 FS $i)}
  print out1 > "aaaaa1"; print out2 > "aaaaa2"; print out3 > "aaaaa3"}' Precipitation.csv

or you can type all the field numbers in the below command. Replace '...' with all the required field numbers

awk '{print $1,$2,$3,...$19,$20 > "aaaaa1";
  print $21,$22,$23,...$39,$40 > "aaaaa2";
  print $41,$42,$43,...$99,$100 > "aaaaa3"}' Precipitation.csv
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.