10

SHORT VERSION (TL;DR)

I have 2 small one-line files, seemingly identical :

$ cat f1 f2
./cconv.sh 100 EUR USD
./cconv.sh 100 EUR USD

But they are not, there is a 1 byte difference in size :

$ ls -l f1 f2
(...) 24 oct.  30 16:19 f1
(...) 23 oct.  30 16:19 f2

$ diff f1 f2
1c1
< ./cconv.sh 100 EUR USD
---
> ./cconv.sh 100 EUR USD

I used dhex to figure out the hexadecimal difference. It appears that :

  • f1 finishes with c2 a0 55 53 44 0a
  • f2 finishes with 20 55 53 44 0a

Does anybody have a clue what's going on here ? What's the difference, and more importantly, where could it come from ? Here is a link to a zip file containing the 2 files, and a screenshot of the dhex result.

LONG VERSION (ADDITIONAL EXPLANATIONS)

The 2 files are excerpts from my ~/.bash_history file.

I noticed a very strange behavior from my shells. Take the following very basic script :

#!/bin/sh
echo $#
echo "$1"
echo "$2"
echo "$3"
exit 0

Under certain circumstance (but which ones ???), it doesn't take the space as an argument separator :

$ ./cconv.sh 100 EUR USD
2
100
EUR USD

But sometimes it works just as it is supposed to :

$ ./cconv.sh 100 EUR USD
3
100
EUR
USD

It drives me nuts ! I spent several hours trying to figure out what's going on. Here are some tests I did to try and narrow it down :

  • I work on a laptop with Debian 11, Gnome 3.38. But I happen to also have a virtual machine with exactly the same OS (D11, G3.38), and in the VM everything works just fine. So obviously I must have done something to my bare metal laptop for it to misbehave. But what ???
  • I noticed that the problem only occurs in a graphical session. If I open a tty (Ctrl+Alt+Fn), it works fine
  • I suspected my terminal emulator. But the behavior is the same in different emulators (I tried Gnome Terminal, Terminator and Konsole, same result)
  • I suspected the shell. But the behavior is the same either with Bash or Dash
  • I disabled all customization I could think of :
    • I temporarily removed /etc/bashrc, /etc/profile, /etc/inputrc and /etc/rc.local
    • I temporarily removed ~/.bashrc, ~/.profile and ~/.inputrc
    • I disabled all Gnome Shell's extensions
  • I even suspected the keyboard, and plugged in a USB keyboard. Same result.

I'm really confused, and have not a clue what's going on. I finally noticed that small difference between the 2 commands in ~/.bash_history : one comes from my Gnome session, the other comes from my tty session. Obviously there's a difference, but what is it exactly, and what could be the cause ?

6
  • 1
    stackoverflow.com/a/2774507/2072269 - C2 A0 = NBSP and 20 would be your normal, plain space. Possibly relevant: unix.stackexchange.com/questions/668563/…
    – muru
    Commented Oct 31, 2022 at 8:53
  • 1
    Many thanks for your hint. Mystery solved ! I indeed configured Shift+Space in Gnome to input NBSP. And it appears that while typing "EUR USD", I kept the Shift key down, hence I had a NBSP instead of a space. The solution is to use another modifier key for NBSP, not Shift. Wow... I spent 2 hours yesterday trying to figure out what was going on...
    – ChennyStar
    Commented Oct 31, 2022 at 9:08
  • od -a is a pretty good tool for checking this king of mixture of characters in a file you expect to be text.
    – Sotto Voce
    Commented Oct 31, 2022 at 20:02
  • @SottoVoce : nice tool. Thanks, never heard of it before. Yes, it's a quick way to find out what's wrong with a string
    – ChennyStar
    Commented Nov 1, 2022 at 8:09
  • Using -v, --show-nonprinting with cat can also be a quick way of finding odd characters in a text file.
    – UncleCarl
    Commented Nov 1, 2022 at 16:17

1 Answer 1

21

c2 a0 is the UTF-8 encoding of the non-breaking space character. It usually looks like a regular space, but isn't recognized as whitespace by the shell.

In a few keymaps, something like AltGr+Space, or Option+Space produces a non-breaking space. Which is amusing if your keymap also has e.g. the pipe character behind AltGr or Option, making it easier to type |<nbsp> instead of |<sp>, giving you errors like this:

$ echo foo | grep .
bash:  grep: command not found

(I think SE folds the nbsp to a regular space, so you probably won't get the error if you copypaste that from here.)

If you copied and pasted the arguments from some other tool, you might get odd formatting from there, but it depends on the program.

See Deal with nbsp character in shell for some solutions for not producing the character.

5
  • 2
    Thanks ! Mystery solved ! I indeed configured Shift+Space in Gnome to input NBSP. And it appears that while typing "EUR USD", I kept the Shift key down, hence I had a NBSP instead of a space. The solution is to use another modifier key for NBSP, not Shift.
    – ChennyStar
    Commented Oct 31, 2022 at 9:12
  • 1
    AltGr seems to be a poor choice too, because of the pipe example you mentioned. I ended up using Shift+AltGr as a modifier for NBSP (it's called 4th level in Gnome's settings).
    – ChennyStar
    Commented Oct 31, 2022 at 9:24
  • @ChennyStar, or just disable it completely if you don't need it... That linked Q&A has a solution for Bash specifically, basically binding the NBSP input to produce a space instead with Bash's bind. I didn't test it, though, so YMMV.
    – ilkkachu
    Commented Oct 31, 2022 at 10:19
  • 2
    It is disabled by default in Gnome, but I specifically activated it, because I needed it in some cases. I just didn't think of the side effects. I think Shift+AltGr+Space is a good compromise. It allows easy access to the NBSP, without having any kind of side effects.
    – ChennyStar
    Commented Oct 31, 2022 at 11:03
  • Also, Ctrl+Shift+Space seems to be the shortcut commonly used for NBSP by office suites (LibreOffice, MS Office, ...).
    – ChennyStar
    Commented Nov 2, 2022 at 8:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .