I'm currently fiddling around with a dying harddisk, and while trying to cp data from it, I get errors like

cp: error reading ‘brokenFile’: Input/output error
# comparison: backtick: `, apostrophe '

I now try to copy the failed files again by using sed to convert the error messages to new cp invocations (I was successfull some times already). But: what are the funny quotation marks? They are even different at start/end. It is no backtick and no apostrophe. I copy/pasted it into my regex, but is there a better way? Maybe using compose?

  • 1
    It probably generates some Unicode characters for English quotes. You should get error messages more friendly to sed (and automated processing in general) if you set the environment variable LANG to C. – celtschk Apr 25 '14 at 14:05
  • They appear to be either wide or unicode characters. This is what i get when I try to put them in my terminal. ` ▒~@~Y ▒~@~X ` – Livinglifeback Apr 25 '14 at 14:07
  • 1
    OK, I now searched for them in the character map. They are U+2018 LEFT SINGLE QUOTATION MARK and U+2019 RIGHT SINGLE QUOTATION MARK. – celtschk Apr 25 '14 at 14:08
  • @celtschk LANG=C works like a charm. I usually have en_US.utf8. – Jasper Apr 25 '14 at 14:19
  • ... and sice I asked for compose key usage: the german wikipedia says Comp+<+' for left and Comp+>+' for right quotation mark. @celtschk do you want to write that up as an answer? – Jasper Apr 25 '14 at 14:24

The characters generated are U+2018 LEFT SINGLE QUOTATION MARK () and U+2019 RIGHT SINGLE QUOTATION MARK (). Those are typographical single quotes for the English language, and are generated because of your current locale. If your current keyboard layout has a compose key, you can enter them with Compose<' (left quote) and Compose>' (right quote).

However, if you want to process the output of a command with other tools (like, in your case, sed), it is usually easier to change the local by setting the environment variable LANG to C. That way, programs will output error messages (and more generally, all output meant to be human-readable) in pure ASCII, which is generally more easily handled using text-based tools.

  • You can set the locale "locally" to each command by just adding "LC_ALL=C" in front of the command itself. ex: LC_ALL=C cp something somewhere . Only cp here will have that locale forced. BUT don't forget to add similar forcing to any interpreting commands as well (sed, etc). [otherwise things like [A-Z] ranges may turn out to be all letters except 'a', for example... read for example gnu.org/software/gawk/manual/html_node/Ranges-and-Locales.html . Added bonus : limiting to C range makes ranges faster, as containing far less characters in them than their unicode counterparts) – Olivier Dulac Apr 25 '14 at 17:15

A useful command to find out about a given character is Free recode (formerly known as GNU recode):

$ echo -n ‘’ | recode ..dump
UCS2   Mne   Description

2018   '6    left single quotation mark
2019   '9    right single quotation mark

Or the unicode command from Debian:

$ unicode ‘’
UTF-8: e2 80 98  UTF-16BE: 2018  Decimal: &#8216;
Category: Pi (Punctuation, Initial quote)
Bidi: ON (Other Neutrals)

UTF-8: e2 80 99  UTF-16BE: 2019  Decimal: &#8217;
Category: Pf (Punctuation, Final quote)
Bidi: ON (Other Neutrals)
  • I get from echo -n ‘ | recode ..dump the following: 1820 lettre mongole a ... – Jasper Apr 25 '14 at 16:07
  • @Jasper, on what system? I can't really see how the 18 and 20 could be revered. What's the output of locale charmap for you? Do you get the same with recode u8..dump? – Stéphane Chazelas Apr 25 '14 at 17:51
  • locale charmap --> UTF-8. Same output with u8..dump. I'm running ArchLinux. – Jasper Apr 25 '14 at 18:03
  • on an x86_64 CPU. I didn't realize until now that my numbers are just swapped... endianness-issue?! – Jasper Apr 25 '14 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.