Sort order depends on your locale, so there's probably some odd interpretation going on there. I get the same results as you on my system. Here, a comma is ignored by the locale. Hence, if we add
296 to your list:
$ sort /tmp/tosort
This also explains the results from your second command. The comma isn't being sorted first, but being ignored. Hence, it's essentially sorting a empty string. If you add a literal empty string to your test, you'll also see it at the beginning.
$ echo -e "2\n,\n3\n" | sort
(It's unclear to me why the empty string precedes the comma. I suspect that the comma is used when there is a tie.)
You will get more predictable results using a "standard" POSIX locale.
$ export LC_ALL=C
$ sort foo