I have a data file, which can have N rows, and each row is composed M elements separated by white space. Currently, I want to separate each row into several segments. In other words, assume the number of segments is 3; then the original file will be separated into 3 files, each of which has N rows and each row has M/3 elements. Besides writing C++ or Java program, Is there any efficient approach that can fulfill this task on Unix/Linux?


This caters for a varying number of fields in the same file, and the last segment being only partialy filled, ie less fields than specified (per segment).
Note though, that if the number of fields in a line results in fewer segments than specified, nothing is written to the output file for those shortfall segments.

awk -v 'ncol=5' -v 'pfix=file' '{
    fldn = 0
    sfix = 1
    segs = NF/ncol
    # round up if number of field is not evenly divisible by number of columns    
    segs = (segs == int(segs)) ?segs :int(segs)+1   
    while (fldn != NF) {
        fmod = (++fldn) % ncol
        printf "%s%s", dlim, $(fldn) >> pfix sfix 
        if (fmod == 1 ) { dlim = " " }
        if ((fmod==0 ) || (fldn==NF))  { 
            printf "\n" >> pfix sfix 
            dlim = ""; sfix++ 
}' infile

Is this what you are looking for?

awk '{ print $1 $2 $3 > file1; print $4 $5 $6 > file2; print $7 $8 $9 > file3 }' originalfile

Or did you want something more general?

awk -v 'n=3' -v 'prefix=pref' '{
    for (i = 0; i < n; i++) {
        for (j = 0; j < NF / n; j++) {
            printf("%s ", $(i + j + 1)) > prefix i
        printf("\n") > prefix i
}' originalfile

Note: this relies on the assumption that all rows have the same number of columns.


If your file is clean, I would advice to use the standard application cut

cut has three flags that you should know at least

  • -d to define the delimiter (TAB is the default
  • -f to select the field
  • -c to select a character range

You can choose to either use the combination -d -f, or -c If your file is not TAB delimited, but nicely delimited by space, you can do

cut -d' ' -f1-3

to select the first three columns.

If you want to select the 4th column, that is in between character 25 and 36, you can do

cut -c25-36





while [[ $SEG_NO -le $SEG ]]
  awk '{CL=NF/'"$SEG"';CL=(CL==int(CL)?CL:int(CL)+1);LS=(('"$SEG_NO"'-1)*CL)+1;LE=LS+CL-1;if(LE>NF)LE=NF;for(i=LS;i<=LE;i++)printf("%s ",$i);printf("\n")}' $FILENAME > ${FILENAME}_$SEG_NO
  SEG_NO=`echo "$SEG_NO + 1"|bc`

Usage: ./sep_file.ksh <file_name_to_read> <no_of_segments>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.