Hot answers tagged

38

When Vim reads an existing file, it tries to detect the file encoding. When writing out the file, Vim uses the file encoding that it detected (except when you tell it differently). So a file detected as UTF-8 is written as UTF-8, a file detected as Latin-1 is written as Latin-1, and so on. By default, the detection process is crude. Every file that you open ...


34

It's your locale and tr problem. Currently, GNU tr fully supports only single-byte characters. So in locales using multibyte encodings, the output can be weird: $ </dev/urandom LC_ALL=vi_VN.tcvn tr -dc '[:print:]' | head -c 64 `�pv���Z����c�ox"�O���%�YR��F�>��췔��ovȪ������^,<H ���> The shell will print multi-byte characters correctly, but GNU ...


33

As noted by others, there isn't really an answer to this: filenames and paths do not have an encoding; the OS only deals with sequence of bytes. Individual applications may choose to interpret them as being encoded in some way, but this varies. Specifically, Glib (used by Gtk+ apps) assumes that all file names are UTF-8 encoded, regardless of the user's ...


24

That's a really nice catch. From a quick look at the source code for GNU find, I would say this boils down to how fnmatch behaves on invalid byte sequences (pred_name_common in pred.c): b = fnmatch (str, base, flags) == 0; (...) return b; This code tests the return value of fnmatch for equality with 0, but does not check for errors; this results in any ...


22

Short answer: restrictions imposed in Unix/Linux/BSD kernel, namei() function. Encoding takes place in user level programs like xterm, firefox or ls. I think you're starting from incorrect premises. A file name in Unix is a string of bytes with arbitrary values. A few values, 0x0 (ASCII Nul) and 0x2f (ASCII '/') are just not allowed, not as part of a ...


21

It is known as carriage return. If you're using vim you can enter insert mode and type CTRL-v CTRL-m. That ^M is the keyboard equivalent to \r. Inserting 0x0D in a hex editor will do the task. How to remove? You can remove it using the command perl -p -i -e "s/\r//g" filename. As the OP suggested in the comments of this answer here, you can even try a ...


20

the file command makes "best-guesses" about the encoding. Use the -i parameter to force file to print information about the encoding. I created two files containing german umlauts. one encoded in utf-8 and one encoded in iso-8859-1. $ file -i * file1: text/plain; charset=utf-8 file2: text/plain; charset=iso-8859-1


20

I think you probably want iconv. It's for converting between codesets and supports an absurd number of formats. For example, to strip anything not valid in UTF-8 you could use: iconv -c -t UTF-8 < input.txt > output.txt Without the -c option it'll report problems in converting to stderr, so with process direction could you save a list of these. ...


18

The char type in C is one byte, but it's intended for ASCII characters; there are variable-width encodings like UTF-8 that can take up many bytes per character. wc uses the mbrtowc(3) function to decode multibyte sequences, depending on the locale set by the LC_CTYPE environment variable. If you set the locale properly, you should get the same result for all ...


17

I guess you see this � invalid character because the name contains a byte sequence that isn't valid UTF-8. File names on typical unix filesystems (including yours) are byte strings, and it's up to applications to decide on what encoding to use. Nowadays, there is a trend to use UTF-8, but it's not universal, especially in locales that could never live with ...


17

You're right about the line endings being important; both OSes expect the line to end with "\n", but Windows also adds a "\r" before that that unix doesn't expect, so unix programs will output the "\r" in their own way. The file doesn't actually end with the two characters "^" and "M", that's just a common way to represent unprintable characters. Programs ...


17

You have to take your codes into quotes: echo -n -e '\x66\x6f\x6f' > byteFileForNow cause otherwise shell replaces \x to x before it goes to echo -e. ps. double escape will also work: echo -n -e \\x66\\x6f\\x6f > byteFileForNow


17

There is no specific character encoding mandated by POSIX. The only character in a fixed position is null, which must be 00. What POSIX does require is that all characters from its Portable Character Set exist. The Portable Character Set contains the printable ASCII characters, space, BEL, backspace, tab, carriage return, newline, vertical tab, form feed, ...


16

Change the character translation in PuTTY to UTF-8.


15

UTF-8 is a variable length encoding of Unicode. It is designed to be superset of ASCII. See Wikipedia for details of the encoding. \x00 \x01 \xF6 \x15 would be UCS-4BE or UTF-32BE encoding. To get from the Unicode code point to the UTF-8 encoding, assuming the locale's charmap is UTF-8 (see the output of locale charmap), it's just: $ printf '\U1F615\n' 😕 $...


14

Edit: I've fixed a typo-bug in the regex.. It needed a '\x80` not \80. The regex to filter out invalid UTF-8 forms, for strict adherance to UTF-8, is as follows perl -l -ne '/ ^( ([\x00-\x7F]) # 1-byte pattern |([\xC2-\xDF][\x80-\xBF]) # 2-byte pattern |((([\xE0][\xA0-\xBF])|([\xED][\x80-\x9F])|([\xE1-\xEC\xEE-\xEF][\x80-\xBF]))([\...


14

It should be pointed out that Mac OS X uses \n a.k.a linefeed (0x0A) now, just like all other *nix systems. Only Mac OS versions 9 and older used \r (CR). Reference: Wikipedia on newlines.


14

The thing is, the kernel doesn't care one bit how the applications interpret the data it is given as a filename. Let's imagine I have a C application that deals with exclusively UTF-16 strings. And I enter, via a properly configured input method, the ∯ symbol (Unicode 0x222F) into the "Save As" prompt/dialog. If the application doesn't do any form of ...


14

At a guess, Your locale uses UTF-8 encoding, and About 10% of your file consists of characters which require more than one octet to encode into UTF-8. By the way, from man wc: -c, --bytes print the byte counts -m, --chars print the character counts


14

Assuming that "foreign" means "not an ASCII character", then you can use find with a pattern to find all files not having printable ASCII characters in their names: LC_ALL=C find . -name '*[! -~]*' (The space is the first printable character listed on http://www.asciitable.com/, ~ is the last.) The hint for LC_ALL=C is required (actually, LC_CTYPE=C and ...


13

Running the locale command will give you information about your locale settings; the character encoding is given by the LC_CTYPE setting. Under Ubuntu, the default locale settings are given in /etc/default/locale. You can change the character encoding by setting LC_CTYPE in your ~/.profile on the VPS, e.g. export LC_CTYPE=en_US.UTF-8 You'll have to make ...


13

If you want to use grep, you can do: grep -axv '.*' file in UTF-8 locales to get the lines that have at least an invalid UTF-8 sequence (this works with GNU Grep at least).


13

find -name option uses shell pattern matching notation to perform matching filename. * is a pattern matching multiple characters, shall match a string of zero or more characters. find uses fnmatch to check pattern matching, so you can use ltrace to check the result: $ touch $'\U1212'aa $ touch D$'\351'sinstaller $ LC_ALL=en_US.utf8 ltrace -e fnmatch find -...


12

It isn't always possible to find out for sure what the encoding of a text file is. For example, the byte sequence \303\275 (c3 bd in hexadecimal) could be ý in UTF-8, or ý in latin1, or Ă˝ in latin2, or 羸 in BIG-5, and so on. Some encodings have invalid byte sequences, so it's possible to rule them out for sure. This is true in particular of UTF-8; most ...


12

Look at this question. Text is usually encoded in UTF-8; so you have to use the hex vales of the bytes used in utf-8 encoding. grep "["$'\xe0\xa4\x85'"-"$'\xe0\xa4\xb5'"]" and grep '[अ-व]' are equivalent, and they perform a locale-based matching (that is, matching is dependent on the sorting rules of devanagari script (that is, the matching is NOT "...


11

Run reset. From the man page: When invoked as reset, tset sets cooked and echo modes, turns off cbreak and raw modes, turns on newline translation and resets any unset special characters to their default values before doing the terminal initialization described above. This is useful after a program dies leaving a terminal in an abnormal state.


11

Your locale information isn't set up correctly on the server machine. Specifically, the LC_CTYPE variable, which indicates the encoding of characters on the terminal, isn't set correctly (or, I suspect, at all). In your local terminal, run locale to see your locale settings. You'll probably see (amongst other lines) something like LC_CTYPE=en_US.UTF-8; what ...


11

In the documentation here, .docx is not listed as a compatible input: Pandoc is a Haskell library for converting from one markup format to another, and a command-line tool that uses this library. It can read markdown and (subsets of) Textile, reStructuredText, HTML, LaTeX, MediaWiki markup, Haddock markup, OPML, and DocBook; and it can write ...


11

file tells you “Non-ISO extended-ASCII text” because it detects that this is: most likely a “text” file from the lack of control characters (byte values 0–31) other than line breaks; “extended-ASCII” because there are characters outside the ASCII range (byte values ≥128); “non-ISO” because there are characters in the 128–159 range (ISO 8859 reserves this ...


11

tr and grep are text processing tools. They expect their input to be text files. Computers store data as sequences of bytes. A text is a sequence of characters. There are several ways to encode characters as bytes, called character encodings. The de facto standard character encoding in most of the world, especially on OSX, is UTF-8, which is an encoding for ...



Only top voted, non community-wiki answers of a minimum length are eligible