5

I wrote a bash script that processes a list of filenames (using glob expansion, as in for f in *) and then outputs a subset of this list to a file. Subsequently I read the content of this file into an array and perform a plain binary search for specific filenames using the obvious < and > operators for comparing strings.

Considering that I would like the script to work on as many different environments like Linux, MacOS, MinGW, ... (even though it uses things like [[ and stat which are less portable), my questions are:

  1. Do I need to sort the file content (either with sort or additional bash code) or is the glob expansion always sorted - in every environment?
  2. Do the conditional operators use the same "sorting" as the expansion (or after sort)?

    Would expansion or sort return file10.txt after file2.txt (in what cases?) but using conditional operators file10.txt would be before file2.txt ? What sort option would I use to fix this?

  3. Are there any caveats if some of my filenames are in Unicode?

  4. Are there any issues using specific versions of bash?
  5. Does LC_COLLATE affect any of the above?

I obviously need the file content to match the sorting "method" of the operators in order for the binary search to work as expected...

  • Glob expansion is about file names, it does not affect "file content" in any way. Maybe you should update your first question. – user79743 Mar 25 '16 at 18:02
  • @BinaryZebra He refers to the file containing the list of files, I believe. – Kusalananda Aug 3 '16 at 18:21
  • 1
    @Kusalananda YesI understand that he sends the result of a globbing operation to a file, and that when he asks about "file contents" he is actually meaning "glob results". But that is a source for misunderstandings: the content of a file could not be sorted by globbing and the result of a glob only affects the content of a file if the user redirects the output of the glob to the file. The point I am trying to make is that: The two elements are different and should be kept apart in our minds. And that they only get mixed by some action of an user. – user79743 Aug 3 '16 at 21:55
2

Yes, glob expansion is always sorted.
In bash (from LESS=+/'^ *Pathname Expansion' man bash)

Pathname Expansion ... the word is regarded as a pattern, and replaced with an alphabetically sorted list of file names matching the pattern.

This is also specified by POSIX glob:

... The pathnames are in sort order as defined by the current setting of the LC_COLLATE category.

Note1: unless the GLOB_NOSORT flag is set. In which case the order is unspecified.

Note2: The sort order is Alphabetic (not numeric), 10 sorts before 2.


Answers:

  1. Do I need to sort the file content (either with sort or additional bash code) ...

Globing has no relation to the file contents, only works with file names.
If you need to sort the "file contents", then, yes, you do need to call sort of use quite a bit more bash code.

  1. ... or is the glob expansion always sorted - in every environment?

Unless it is disabled with GLOB_NOSORT the result of Globing is sorted in the order defined by the collation order (variable LC_COLLATE) in the environment.

To have the same sort order you must have the same collation in effect. Both setting a LC_COLLATE variable and having a locale description that contains the same collate details.

  1. Do the conditional operators use the same "sorting" as the expansion (or after sort)?

Yes. Both are affected in the same way by LC_COLLATE.

  1. Would expansion or sort return file10.txt after file2.txt (in what cases?) but using conditional operators file10.txt would be before file2.txt ? What sort option would I use to fix this?

A result of 10 before 2 is "dictionary order" which is the same as what is called "alphabetic order" in the bash manual description. So, if you use bash (or any POSIX shell) to sort, that's the order you will get (in all cases). That's not wrong, so it is not fixable (for text).

However, if you choose to use sort (an external tool, outside the shell) you may ask for numeric sort (the -n option), which will place 2 before 10. Or you may extract the numbers from the text and use them to make an integer comparison (the -lt -gt integer operators) in the shell.

Are there any caveats if some of my filenames are in Unicode?

Mostly: Collation order is not fixed.

It changes with time and UNICODE version.

What may happen is that you get some surprising results in some language that you are not familiar with. For example:

"aa" would match "å" in a Danish

In short: » Be prepared to be surprised «.

Are there any issues using specific versions of bash?

Well, you must use a bash version above 2.0

respect LC_COLLATE  2.0 

Does LC_COLLATE affect any of the above?

The variable LC_COLLATE affect all of the above.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.