separate a file into several small files according to columns

Question

I have a data file, which can have N rows, and each row is composed M elements separated by white space. Currently, I want to separate each row into several segments. In other words, assume the number of segments is 3; then the original file will be separated into 3 files, each of which has N rows and each row has M/3 elements. Besides writing C++ or Java program, Is there any efficient approach that can fulfill this task on Unix/Linux?

Answer 1

This caters for a varying number of fields in the same file, and the last segment being only partialy filled, ie less fields than specified (per segment).
Note though, that if the number of fields in a line results in fewer segments than specified, nothing is written to the output file for those shortfall segments.

awk -v 'ncol=5' -v 'pfix=file' '{
    fldn = 0
    sfix = 1
    segs = NF/ncol
    # round up if number of field is not evenly divisible by number of columns    
    segs = (segs == int(segs)) ?segs :int(segs)+1   
    while (fldn != NF) {
        fmod = (++fldn) % ncol
        printf "%s%s", dlim, $(fldn) >> pfix sfix 
        if (fmod == 1 ) { dlim = " " }
        if ((fmod==0 ) || (fldn==NF))  { 
            printf "\n" >> pfix sfix 
            dlim = ""; sfix++ 
        }
    } 
}' infile

Answer 2

If your file is clean, I would advice to use the standard application cut

cut has three flags that you should know at least

• -d to define the delimiter (TAB is the default
• -f to select the field
• -c to select a character range

You can choose to either use the combination -d -f, or -c If your file is not TAB delimited, but nicely delimited by space, you can do

cut -d' ' -f1-3

to select the first three columns.

If you want to select the 4th column, that is in between character 25 and 36, you can do

cut -c25-36

Answer 3

Is this what you are looking for?

awk '{ print $1 $2 $3 > file1; print $4 $5 $6 > file2; print $7 $8 $9 > file3 }' originalfile

Or did you want something more general?

awk -v 'n=3' -v 'prefix=pref' '{
    for (i = 0; i < n; i++) {
        for (j = 0; j < NF / n; j++) {
            printf("%s ", $(i + j + 1)) > prefix i
        }
        printf("\n") > prefix i
    }
}' originalfile

Note: this relies on the assumption that all rows have the same number of columns.

Answer 4

sep_file.ksh

#!/bin/ksh

FILENAME=$1
SEG=$2

SEG_NO=1

while [[ $SEG_NO -le $SEG ]]
do
  awk '{CL=NF/'"$SEG"';CL=(CL==int(CL)?CL:int(CL)+1);LS=(('"$SEG_NO"'-1)*CL)+1;LE=LS+CL-1;if(LE>NF)LE=NF;for(i=LS;i<=LE;i++)printf("%s ",$i);printf("\n")}' $FILENAME > ${FILENAME}_$SEG_NO
  SEG_NO=`echo "$SEG_NO + 1"|bc`
done

Usage: ./sep_file.ksh <file_name_to_read> <no_of_segments>