Last line is wrong when splitting a file with awk

Question

I have a data file main.txt that looks like this

20130826,aaaaaaaaaaaaaa,bbbbbbb
20130826,sdfasdfasdfas,sdfasdfasd
20130826,dfasdfas,asdf2323
...
20130827,sfasdfasdfasd,sdfasdfwea

I used the following awk snippet to split main.csv into smaller files by date.

cat test01 | stdbuf -oL -eL awk -F',' '{print $2","$3 >> "data"$1".csv"; fflush()}'

and I found out the for the splited file data20130826.csv the last line is incomplete,

...
20130826,dfasdfas,asdf2323
2013082

in fact the last few lines(10 or so) of date 20130826 are all missing, they are not in data20130827.csv either. I tried to turn off buffering as in the above line, but it doesn't seems to help. I am sure the main file is OK. What's going wrong? I'm using GNU Awk 4.0.1.

Answer 1

Just to clarify my original post: I agree with Orion and doubt that this is an awk bug. I also wouldn't bother with fflush. I think that this is just an awk problem caused by data. Specifically I think you may have too many open file descriptors - one for each date in your file. Or perhaps you've reached a buffer limit for the same reason - too many open files. fflush doesn't close opened files - it just flushes buffers.

So how many different dates are in the input file? Thats how many open files you'll have:

cut -d"," -f1 test01 | sort | uniq | wc -l

- if it's a lot (hundreds) then see my 2nd suggestion regarding closing files as you go.

Two suggestions:
1. Are you sure the data is just plain text without any hidden characters like backspaces and also has normal UNIX line endings ? Can you grep out the lines prefixed "20130826" to see that they look normal and really are all separate lines, ie

cat test01 | grep "^20130826"
- and also run
cat test01 | grep -c "^20130826"

- to confirm that grep line count matches what's in the output file (or not)

2. If the input data is sorted in date order then you could try closing the files as you write them: I tested this since my original post and it worked fine:

cat test01 | awk -F"," '{prevfile=ofile; ofile=sprintf("data%s.csv",$1);
             if (NR > 1 && ofile != prevfile) close(prevfile); print $2","$3 >> ofile}'

This code will still work if your file isn't sorted by date but will open and close files more often. In that case just change "cat test01" to "sort test01" at the start of the command.

You don't always need to close files explicitly when using awk but I know from experience that awk used to crash if you wrote a lot of files like this without closing any of them It's possible that still applies as I think it relates to a limit on open file descriptors.

Also as you're appending to these files make sure that they don't contain any data before you run the command. Easy to forget when things aren't working...

Answer 2

Try:

$ cat test01 |
stdbuf -oL -eL awk -F',' '{print $2","$3 >> "data"$1".csv"; fflush("")}'

In gawk 4.0.1, calling fflush() without any argument only flush stdout. You need calling fflush() with empty string "" to make awk flushes all open output files and pipes as well.

With gawk 4.0.2 and later, if there is no argument, or if the argument for fflush() is the null string "", then awk flushes the buffers for all open output files and pipes.