2

I have a file with 10,671 columns.

I am trying to split it into 10 smaller files, each with 1000 columns until all columns have been printed, named file_transposed01, file_transposed02, ... file_transposed10. I am printing the first column of the file as the first column for every new sub-file.

I know it is possible to do this by row number using awk.

I am trying to adapt this approach using NF instead of NR to split by columns:

awk 'BEGIN{$1;}NF%1000==2{x="masterfile_transposed"++i;a[i]=x;print f>x;}{print > x}' masterfile

But it gives an error:

expression for `>' redirection has null string value. I am not sure what is wrong with my syntax. Is it possible to do this task using awk?

How can I make this approach work, or is there a better approach to take?

2 Answers 2

1
awk -v ncols=1000 '
    {
        f=1
        file = "file_" f
        for (i=1; i <= NF; i++) {
            printf "%s%s", $i, OFS > file
            if (i % ncols == 0) {
                print "" > file
                f++
                file = "file_" f
            }
        }
        print "" > file
    }
' file
1

I dunno what you're delimiting these columns on, but I made a file like:

 i=0
 until [ "$((i+=1))" -gt 100 ]
 do    seq -s '     ' 10671 
 done >/tmp/file

Which resulted in 100 lines at 10671 columns a piece delimited on sequences of spaces.

I next did:

sed 's/  */\n/1000;/\n/P;//D;G' </tmp/file | sed 's/.* //'

I used the second sed because I wanted to clearly see where the data split - it can be a little difficult to make much sense of one-thousand columns on a single line. So I elide each line to its final column. Every time that I use s/.* // here it is because I want to show only the final column for a line. For example, when I first ran this it printed...

1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
10671

...100 times. And this means that each line actually looks like...

1   2  3...1000
1001 1002 1003...2000

...and so on. sed was definitely splitting the data correctly...

So I figured it should be really easy to split that out into separate files by line. So my next step was:

sed 's/  */\n/1000;/\n/P;//D;G' </tmp/file | 
sed -n "$(printf 'w /tmp/outfile.%d\nn\n' {1..11})"

...and I wound up with 11 files in /tmp in which all of the columns were split out separately. I can do...

sed 's/.* //' /tmp/outfile.1

... and it will print like...

1000
1000
1000
1000

...25 times. or...

sed 's/.* //' /tmp/outfile.11

...and...

10671
10671
10671
10671

...also 25 times...

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .