3

Let's say I have a directory with the files _b, a, c, č, d. I would like to sort the files according to the cs_CZ.UTF8 locale but without ignoring the underscore, ie like this: _b a c č d.

Currently, ls (and ls | sort as well) sorts the files like this: a _b c č d. All answers I have found suggest using LC_COLLATE=C, but that changes the ordering to this: _b a c d č (notice that č is now at the end, not between c and d as it is supposed to be).

Is there any way to achieve this goal?

Note that I also care about other characters than underscore, ie I would like a-n.pdf a-p.pdf a.pdf c č d sort in this order and not a-n.pdf a.pdf a-p.pdf c č d. (EDIT: Actually, a.pdf a-n.pdf a-p.pdf c č d is also fine, as long as the non-alphanumeric characters are not ignored.)

The following are not the answers I am looking for:

  • using LC_COLLATE=C as explained above,
  • using shell expansion such as ls _*; ls [^_]* because the question is not only about underscores.
6
  • That's an interesting challenge; it might be helpful to either copy, or move your sample data / expected results into a table to make it a little easier to understand how hyphens and underscores should be handled. The help page unix.stackexchange.com/editing-help#tables shows the markdown for tables. Oct 13, 2021 at 22:46
  • When you edit your question, you will probably want to include the [locale] tag also. Have you looked at unix.stackexchange.com/q/421908/90290 and unix.stackexchange.com/q/39827/90290 yet? Oct 13, 2021 at 23:37
  • @MarkStewart I don't care exactly whether how exactly non-alphanumeric characters are ordered as long as they are not ignored completely. Thus a.pdf a-n.pdf a-p.pdf is OK, as is a-n.pdf a-p.pdf a.pdf, but not a-n.pdf a.pdf a-p.pdf as that clearly ignores the hyphens/periods. I have tried to clarify this in the question. Oct 14, 2021 at 13:58
  • @MarkStewart Thanks for the pointers, I understand there is no easy solution and I would have to learn how to write my own LC_COLLATE rules. I might do that once I have the free time… It seems strange, however, that no one's ever thought about this before (even though this is how collation works e.g. in Windows Explorer). Oct 14, 2021 at 14:09
  • Also, this might be a case of the Mandela effect, but I think I remember the sorting in ls working like this in some Linux distros some 10–15 years ago. Oct 14, 2021 at 14:11

1 Answer 1

2

On a GNU system, appending NULs to non-alphas may help:

$ ls | sed 's/[^[:alpha:]]/&\x0/g' | sort | tr -d '\0'
_b
a
c
č
d

That assumes file names don't contain newline characters. Generally, you can't sort lists of file names with sort as file names can very well be made of several lines themselves.

You could replace the newlines within filenames with / here before sorting. With zsh:

print -rNC1 -- *(N) | # print NUL-delimited
  tr '\n\0' '/\n' |
  sed 's/[^[:alpha:]]/&\x0/g' |
  sort |
  tr -d '\0' |
  tr '/' '\n'

Or to keep the list NUL-delimited so it can be post-processed:

print -rNC1 -- *(N) | # print NUL-delimited
  tr '\n\0' '/\n' |
  sed 's/[^[:alpha:]]/&\x0/g' |
  sort |
  tr -d '\0' |
  tr '/\n' '\n\0'

The strcoll() API that is used for sorting takes two NUL-terminated strings. Traditional sort implementations only support text input and text input precludes NULs so they're fine. GNU sort however, like GNU implementations of most standard text utilities do support NULs with their input.

I don't know how exactly GNU sort handles lines with NULs, but my guess would be that it breaks the lines on NULs and compares segments one to one. So, when comparing foo_\0car with foobar_\0more for instance, foo_ is compared with foobar_ and comes first.

You can also define the order of zsh globs using arbitrary transformations with the oe (order based on the evaluation of some code) or o+function glob qualifiers. But before calling strcoll(), zsh removes the NULs, so you can't use the same transformation as used for GNU sort above.

Instead, there, you could add 0s before sequences of non-alphas and 1s before sequences of alphas:

In ~/.zshrc

set -o extendedglob
mysort() {
  REPLY=${REPLY//(#m)[^[:alpha:]]##/0$MATCH}
  REPLY=${REPLY//(#m)[[:alpha:]]##/1$MATCH}
}

Then:

$ print -rC1 -- *(No+mysort)
_b
a
c
č
d
3
  • Boy, that is a lot easier than a possible solution I had spinning in my head, of adding a "sorting" column to the output of ls, that would hide non-alphanumeric characters, pipe to sort, then pipe to awk to omit the sorting column! Oct 14, 2021 at 14:28
  • This looks cool. Could you please explain how this works a bit more? What is the role of NULs in sorting? Oct 14, 2021 at 14:50
  • @NikolaBenes, see edit. Oct 14, 2021 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.