1

I've tried:

  1. awk '{if (last != $1) close(last); print > $1; last = $1}' file

  2. awk -F$'\t' '{ print > ($1) }' file

  3. awk '{if (last != $1) close(last); print >> $1; last = $1}' file

To split a very large text file (33GB) into multiple files named by first column.

For smaller files everything works fine but for large files awk stops near the end of column type (commands 1 and 2) or forgets to input newline characters for columns that have "." in them (command 3).

Example: it just stops before reaching real end of column of type "10"

10      69331427        1
10      69331428        1
10      69331429        1
10      69331430        1
10      69331431        1
10

EDIT : Closing the file seems to help. '{print >> $1; close($1)}'

GNU Awk 4.1.4, API: 1.1 (GNU MPFR 4.0.1, GNU MP 6.1.2)

8
  • Fixed the first one. Closing the file does seem to help. All three commands are approved answers in different questions regarding the same problem. That's why I found it odd that it creates 99% proper files and then just cut in the end.
    – osowiecki
    Aug 30, 2019 at 19:30
  • Btw, if you use """awk '{print >> $1; close($1)}' testing""" with testing containg """AAEX03026070.1 1676 0 AAEX03026070.1 1677 0""" (tab separated) it will look different in midnight commander (mc) view mode F3 in raw and parsed mode. I belive this might be a bug. The file has normal "\n" characters but mc ignores them for some reason. This happens only when first column is not a number and has a ".".
    – osowiecki
    Aug 30, 2019 at 19:36
  • idk what "midnight commander" is but if it's some kind of text editor and if by "it" in your statement it will look different... you mean the input file - maybe that's an indication that your input file contains undesirable control characters?
    – Ed Morton
    Aug 30, 2019 at 22:01
  • 1
    So it sounds like you have 2 different tools (awk and mc) that are both exhibiting unexpected behavior when operating on your input file. Look to your input file....
    – Ed Morton
    Aug 30, 2019 at 22:38
  • 1
    You should never use {print > $1} without checking if $1 is a proper as a file name (no /, /../, NUL bytes) and without closing it afterwards -- even if GNU awk closes files automatically, that will keep thousands (ulimit -n) of open files around, putting pressure on your system and triggering bugs.
    – user313992
    Aug 30, 2019 at 23:37

1 Answer 1

Reset to default
2

To "To split a very large text file (33GB) into multiple files named by first column." using GNU awk on any UNIX box is this:

awk '{print > $1}' file

That's all. If you're running into problems then it's something outside of your awk command that's causing it, e.g. maybe you're running out of space on your drive or maybe your input file contains some weird control characters.

I don't know what you mean by awk stops near the end of column type, nor forgets to input newline characters for columns that have "." in them, nor it just stops before reaching real end of column of type "10". That may partially be because there's nothing in your question to indicate what "column type" means to you.

15
  • For a very large file with thousands of unique values in the first column, this may cause awk to run out of available file descriptors. This is probably why they try to deal with that by explicitly closing the file when needed.
    – Kusalananda ♦
    Aug 30, 2019 at 21:52
  • @Kusalananda No, it won't. Gawk handles that internally, closing and re-opening files as necessary. It WOULD be a problem with some other awk but the OP is using gawk 4.1.4.
    – Ed Morton
    Aug 30, 2019 at 21:54
  • You appear to be correct. So it may be an unnecessary precaution on the user's side. However I also read that "gawk’s ability to do this depends upon the facilities of your operating system, so it may not always work. It is therefore both good practice and good portability advice to always use close() on your files when you are done with them." (from here) This may not be an issue on Linux though.
    – Kusalananda ♦
    Aug 30, 2019 at 22:05
  • @EdMorton then you should write gawk instead of awk in your answer and mention that. But even with gawk, that is horribly dangerous, 1st because $1 may contain ../.., etc, and 2nd because gawk will accept NUL bytes in its input (which are invisible on terminal), but silently ignore & truncate them when using them in a filename printf 'foo\0bar baz' | gawk '{print$1>$1}'.
    – user313992
    Aug 30, 2019 at 22:05
  • 1
    @mosvy no, there's no need to write gawk when you specifically state "using GNU awk..." and no it's not horribly dangerous - you don't need to protect against /s and NULs and other things when the posted sample input simply has all integers in that field.
    – Ed Morton
    Aug 30, 2019 at 22:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .