I've tried:
awk '{if (last != $1) close(last); print > $1; last = $1}' file
awk -F$'\t' '{ print > ($1) }' file
awk '{if (last != $1) close(last); print >> $1; last = $1}' file
To split a very large text file (33GB) into multiple files named by first column.
For smaller files everything works fine but for large files awk
stops near the end of column type (commands 1 and 2) or forgets to input newline characters for columns that have "." in them (command 3).
Example: it just stops before reaching real end of column of type "10"
10 69331427 1
10 69331428 1
10 69331429 1
10 69331430 1
10 69331431 1
10
EDIT :
Closing the file seems to help.
'{print >> $1; close($1)}'
GNU Awk 4.1.4, API: 1.1 (GNU MPFR 4.0.1, GNU MP 6.1.2)
it will look different...
you mean the input file - maybe that's an indication that your input file contains undesirable control characters?{print > $1}
without checking if$1
is a proper as a file name (no/
,/../
, NUL bytes) and without closing it afterwards -- even if GNU awk closes files automatically, that will keep thousands (ulimit -n
) of open files around, putting pressure on your system and triggering bugs.