Why awk does not show the correct record length?

Question

I'm concerned regarding what awk shows as the record length. I'm checking some files for a specific record length - awk shows the result I wanted, but the file size shows that each record in the file is actually larger than what awk says by 1 byte.

$ ls -l some_file.txt
-rw-r--r--    1 foo   bar           250614 Oct 20 08:49 some_file.txt

$ awk '{ print length }' some_file.txt | sort -u
458

$ echo "(250614%458)" | bc
88

$ echo "(250614%459)" | bc
0

Notice that the bc result is wrong with a record length of 458, but seems fine with a record length of 459. Also, awk + sort shows that all records have a record length of 458. My educated guess is that awk is not accounting for the End Of Line character, hence making a real record length of 459. What do you think?

ps: awk on AIX 5.3

Answer 1

This is because the default Record Separator RS is set to newline.

Therefore awk will interpret this as a separator instead of a character in the length.

To check what RS is set to:

echo | awk '{print "\""RS"\""}'
"
"

The quotes are seperated by a newline showing the RS value.

To confirm that the RS character is not included in the length output:

$ echo test > some_file.txt
$ ls -l
-rw-r--r--. 1 user user    5 Oct 20 16:33 some_file.txt

Show the length with RS set to newline.

$ awk '{print length}' some_file.txt
4

Set RS to be a character that does not exist in the file and count again:

$ awk 'BEGIN {RS=":"} {print length}' some_file.txt
5

The additional character is now included.

Answer 2

What you're seeing is perfectly normal. By default, awk does not include the newline character in a record.

From the POSIX standard for awk:

Input shall be interpreted as a sequence of records. By default, a record is a line, less its terminating <newline>
...
String Functions
   length[([s])] - Return the length, in characters, of its argument taken as a string, or of the whole record, $0, if there is no argument.