Space not taken as an argument separator by shell script (could someone please explain that small file difference ?)

Question

SHORT VERSION (TL;DR)

I have 2 small one-line files, seemingly identical :

$ cat f1 f2
./cconv.sh 100 EUR USD
./cconv.sh 100 EUR USD

But they are not, there is a 1 byte difference in size :

$ ls -l f1 f2
(...) 24 oct.  30 16:19 f1
(...) 23 oct.  30 16:19 f2

$ diff f1 f2
1c1
< ./cconv.sh 100 EUR USD
---
> ./cconv.sh 100 EUR USD

I used dhex to figure out the hexadecimal difference. It appears that :

• f1 finishes with c2 a0 55 53 44 0a
• f2 finishes with 20 55 53 44 0a

Does anybody have a clue what's going on here ? What's the difference, and more importantly, where could it come from ? Here is a link to a zip file containing the 2 files, and a screenshot of the dhex result.

LONG VERSION (ADDITIONAL EXPLANATIONS)

The 2 files are excerpts from my ~/.bash_history file.

I noticed a very strange behavior from my shells. Take the following very basic script :

#!/bin/sh
echo $#
echo "$1"
echo "$2"
echo "$3"
exit 0

Under certain circumstance (but which ones ???), it doesn't take the space as an argument separator :

$ ./cconv.sh 100 EUR USD
2
100
EUR USD

But sometimes it works just as it is supposed to :

$ ./cconv.sh 100 EUR USD
3
100
EUR
USD

It drives me nuts ! I spent several hours trying to figure out what's going on. Here are some tests I did to try and narrow it down :

• I work on a laptop with Debian 11, Gnome 3.38. But I happen to also have a virtual machine with exactly the same OS (D11, G3.38), and in the VM everything works just fine. So obviously I must have done something to my bare metal laptop for it to misbehave. But what ???
• I noticed that the problem only occurs in a graphical session. If I open a tty (Ctrl+Alt+Fn), it works fine
• I suspected my terminal emulator. But the behavior is the same in different emulators (I tried Gnome Terminal, Terminator and Konsole, same result)
• I suspected the shell. But the behavior is the same either with Bash or Dash
• I disabled all customization I could think of :
  • I temporarily removed /etc/bashrc, /etc/profile, /etc/inputrc and /etc/rc.local
  • I temporarily removed ~/.bashrc, ~/.profile and ~/.inputrc
  • I disabled all Gnome Shell's extensions
• I even suspected the keyboard, and plugged in a USB keyboard. Same result.

I'm really confused, and have not a clue what's going on. I finally noticed that small difference between the 2 commands in ~/.bash_history : one comes from my Gnome session, the other comes from my tty session. Obviously there's a difference, but what is it exactly, and what could be the cause ?

Answer 1

c2 a0 is the UTF-8 encoding of the non-breaking space character. It usually looks like a regular space, but isn't recognized as whitespace by the shell.

In a few keymaps, something like AltGr+Space, or Option+Space produces a non-breaking space. Which is amusing if your keymap also has e.g. the pipe character behind AltGr or Option, making it easier to type |<nbsp> instead of |<sp>, giving you errors like this:

$ echo foo | grep .
bash:  grep: command not found

(I think SE folds the nbsp to a regular space, so you probably won't get the error if you copypaste that from here.)

If you copied and pasted the arguments from some other tool, you might get odd formatting from there, but it depends on the program.

See Deal with nbsp character in shell for some solutions for not producing the character.