File splitting Based on Column Name With `awk` Fails

Question

I've tried:

1. awk '{if (last != $1) close(last); print > $1; last = $1}' file

2. awk -F$'\t' '{ print > ($1) }' file

3. awk '{if (last != $1) close(last); print >> $1; last = $1}' file

To split a very large text file (33GB) into multiple files named by first column.

For smaller files everything works fine but for large files awk stops near the end of column type (commands 1 and 2) or forgets to input newline characters for columns that have "." in them (command 3).

Example: it just stops before reaching real end of column of type "10"

10      69331427        1
10      69331428        1
10      69331429        1
10      69331430        1
10      69331431        1
10                 

EDIT : Closing the file seems to help. '{print >> $1; close($1)}'

GNU Awk 4.1.4, API: 1.1 (GNU MPFR 4.0.1, GNU MP 6.1.2)

Answer 1

To "To split a very large text file (33GB) into multiple files named by first column." using GNU awk on any UNIX box is this:

awk '{print > $1}' file

That's all. If you're running into problems then it's something outside of your awk command that's causing it, e.g. maybe you're running out of space on your drive or maybe your input file contains some weird control characters.

I don't know what you mean by awk stops near the end of column type, nor forgets to input newline characters for columns that have "." in them, nor it just stops before reaching real end of column of type "10". That may partially be because there's nothing in your question to indicate what "column type" means to you.