Sort on substring of line (not using a delimiter)

Question

I need some help understanding how the sort command is working in the following examples:

cat testfile

The input data:

thisfilehasduplicates
thissnowhasfallen
thisduckhasdied
thishallwasfull
berthammwasaclown
fredsimmisprimeminister
fredalbaisabinman
janetyceisscottish
janeouseisenglish
janellyriswelsh

The aim here is to sort and deduplicate by the first four characters. I have read that -k1.1,1.4 will achieve this by sorting by the first four characters of the first field. The comma denotes a range of characters with 1.4 meaning, field 1, character 4.

sort -u -t -k1.1,1.4 testfile

I interpret this version as -t followed by blank to mean that there is no field separator, however I suppose it could mean field separator is whitespace, however it somehow interferes with the -k and -u flags and doesn't give me what I want:

berthammwasaclown
fredalbaisabinman
fredsimmisprimeminister
janellyriswelsh
janeouseisenglish
janetyceisscottish
thisduckhasdied
thisfilehasduplicates
thishallwasfull
thissnowhasfallen

sort -u -k1.1,1.4 testfile

This version, without the -t flag does give exactly what is required, deduplication on the first four characters, at least I think that's what it is doing

berthammwasaclown
fredalbaisabinman
janellyriswelsh
thisduckhasdied

I've read the man pages on my distribution (SunOS 5.10), but I don't fully understand the sections relating to the -k and -t flags, especially when using the dot notation in a key specification.

Answer 1

sort -u -t -k1.1,1.4 testfile
I interpret this version as -t followed by blank to mean that there is no field separator, however I suppose it could mean field separator is whitespace,

Neither of those. The tokenization of arguments passed to a program on Unix is defined by the shell, not the program; and in most Unix programs since forever, if (a leading argument which is) an option/flag requires a value and you don't supply it as part of the same argument, the next argument is used as the value. -t -k1.1,1.4 uses -k1.1,1.4 as the field separator, truncating it to one character namely -. As a result there is no -k option recognized, so it sorts using the default key definition, which is the whole line.

GNU sort in this case gives an error, saying that 'tab' value (i.e. the value of -t) is multi-character; this behavior is more helpful, but not required by either tradition or standard. On SunOS-aka-Solaris, utilities are not required to and many actually do not conform to XPG-and-POSIX unless you specify the xpg4 or xpg6 version -- did you? -- but on my (now virtualized) Solaris 10 test system, both /usr/bin/sort and /usr/xpg4/bin/sort have this less-helpful behavior.

If you did want the space character as the field separator -- which would be meaningless for your data, which contains no space characters -- you need to tell the shell to pass a space character either as part of the -t argument or as an additional argument:

sort -t" " -k1.1,1.4
sort -t' ' -k1.1,1.4
sort -t\  -k1.1,1.4
sort "-t " -k1.1,1.4
sort '-t ' -k1.1,1.4

sort -t " " -k1.1,1.4
sort -t ' ' -k1.1,1.4
sort -t \  -k1.1,1.4

Note that in Unix-type systems (whether or not Unix-trademark branded, which SunOS originally wasn't) 'whitespace' usually does not mean only the space character; it usually includes at least the space and (horizontal) tab characters, and often 'vertical' space like LF, VT, FF as well. Standard sort can use only a single character as the field separator, so that cannot be whitespace. In contrast shell, and awk (by default), do parse fields using multiple whitespace characters.