Tag Info

Hot answers tagged

19

When Vim reads an existing file, it tries to detect the file encoding. When writing out the file, Vim uses the file encoding that it detected (except when you tell it differently). So a file detected as UTF-8 is written as UTF-8, a file detected as Latin-1 is written as Latin-1, and so on. By default, the detection process is crude. Every file that you open ...


17

As noted by others, there isn't really an answer to this: filenames and paths do not have an encoding; the OS only deals with sequence of bytes. Individual applications may choose to interpret them as being encoded in some way, but this varies. Specifically, Glib (used by Gtk+ apps) assumes that all file names are UTF-8 encoded, regardless of the user's ...


17

The char type in C is one byte, but it's intended for ASCII characters; there are variable-width encodings like UTF-8 that can take up many bytes per character. wc uses the mbrtowc(3) function to decode multibyte sequences, depending on the locale set by the LC_CTYPE environment variable. If you set the locale properly, you should get the same result for all ...


15

You're right about the line endings being important; both OSes expect the line to end with "\n", but Windows also adds a "\r" before that that unix doesn't expect, so unix programs will output the "\r" in their own way. The file doesn't actually end with the two characters "^" and "M", that's just a common way to represent unprintable characters. Programs ...


14

I guess you see this � invalid character because the name contains a byte sequence that isn't valid UTF-8. File names on typical unix filesystems (including yours) are byte strings, and it's up to applications to decide on what encoding to use. Nowadays, there is a trend to use UTF-8, but it's not universal, especially in locales that could never live with ...


13

At a guess, Your locale uses UTF-8 encoding, and About 10% of your file consists of characters which require more than one octet to encode into UTF-8. By the way, from man wc: -c, --bytes print the byte counts -m, --chars print the character counts


10

Run reset. From the man page: When invoked as reset, tset sets cooked and echo modes, turns off cbreak and raw modes, turns on newline translation and resets any unset special characters to their default values before doing the terminal initialization described above. This is useful after a program dies leaving a terminal in an abnormal state.


10

Short answer: restrictions imposed in Unix/Linux/BSD kernel, namei() function. Encoding takes place in user level programs like xterm, firefox or ls. I think you're starting from incorrect premises. A file name in Unix is a string of bytes with arbitrary values. A few values, 0x0 (ASCII Nul) and 0x2f (ASCII '/') are just not allowed, not as part of a ...


10

Assuming that "foreign" means "not an ASCII character", then you can use find with a pattern to find all files not having printable ASCII characters in their names: LC_ALL=C find . -name '*[! -~]*' (The space is the first printable character listed on http://www.asciitable.com/, ~ is the last.) The hint for LC_ALL=C is required (actually, LC_CTYPE=C and ...


9

tput sgr0, tput rmacs, or tput reset. reset is usually but not always even more complete than tput reset. (Turn off all special output modes; turn off alternate character set, which is usually but not always included in sgr0; send terminal reset string which often does those and more.) It can be useful to embed tput sgr0 at the beginning of $PS1.


8

I think you probably want iconv. It's for converting between codesets and supports an absurd number of formats. For example, to strip anything not valid in UTF-8 you could use: iconv -c -t UTF-8 < input.txt > output.txt Without the -c option it'll report problems in converting to stderr, so with process direction could you save a list of these. ...


8

Running the locale command will give you information about your locale settings; the character encoding is given by the LC_CTYPE setting. Under Ubuntu, the default locale settings are given in /etc/default/locale. You can change the character encoding by setting LC_CTYPE in your ~/.profile on the VPS, e.g. export LC_CTYPE=en_US.UTF-8 You'll have to make ...


8

The thing is, the kernel doesn't care one bit how the applications interpret the data it is given as a filename. Let's imagine I have a C application that deals with exclusively UTF-16 strings. And I enter, via a properly configured input method, the ∯ symbol (Unicode 0x222F) into the "Save As" prompt/dialog. If the application doesn't do any form of ...


8

With bash, zsh, GNU echo or some implementations of ksh on some systems, this can be decoded simply by echo -e after replacing all % with \x. url_encoded_string="%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" temp_string=${url_encoded_string//%/\\x} printf '%s\n' "$temp_string" # output: \xD1\x80\xD0\xB5\xD1\x81\xD1\x83\xD1\x80\xD1\x81\xD1\x8B echo -e ...


7

· in latin1 is the byte sequence c2 b7. That same byte sequence interpreted in UTF-8 is the character ·. So what's happening is that Emacs believes your terminal displays UTF-8 (Unicode) but your terminal actually displays latin1. PuTTY supports UTF-8, so the easiest solution is to tell it to use UTF-8, at least when connecting to this host. As an added ...


6

It is a double byte format, probably UTF-16. See if you can identify a BOM (Byter Order Mark) header at the beginning of the file. This will tell you the encoding if it is present, although it may not be. Note that a text editor may hide this from you, so you will probably need to look at the file with a hex dump utility such as od or something similar to ...


6

Your locale information isn't set up correctly on the server machine. Specifically, the LC_CTYPE variable, which indicates the encoding of characters on the terminal, isn't set correctly (or, I suspect, at all). In your local terminal, run locale to see your locale settings. You'll probably see (amongst other lines) something like LC_CTYPE=en_US.UTF-8; what ...


6

It isn't always possible to find out for sure what the encoding of a text file is. For example, the byte sequence \303\275 (c3 bd in hexadecimal) could be ý in UTF-8, or ý in latin1, or Ă˝ in latin2, or 羸 in BIG-5, and so on. Some encodings have invalid byte sequences, so it's possible to rule them out for sure. This is true in particular of UTF-8; most ...


6

For external representations, UTF-8 is definitely the standard. Some 8-bit encodings are still strong (mostly in Europe) and some 16-bit encodings are still strong (mostly in East Asia), but they are clearly legacy encodings, on their slow way out. UTF-8 is standard not only on unix, but also on the web. For internal representations, there's no such ...


6

You could use the iconv character set conversion GNU utility. Here's an example: iconv --from-code=UTF-8 --to-code=MAC INPUTFILE > OUTPUTFILE where INPUTFILEand OUTPUTFILE are the names of your original and converted file (which may be the same). You'd alaso need to substitute UTF-8 by your native encoding, if it's not Unicode, and MAC may not be ...


5

the file command makes "best-guesses" about the encoding. Use the -i parameter to force file to print information about the encoding. I created two files containing german umlauts. one encoded in utf-8 and one encoded in iso-8859-1. $ file -i * file1: text/plain; charset=utf-8 file2: text/plain; charset=iso-8859-1


5

Edit: I've fixed a typo-bug in the regex.. It needed a '\x80` not \80. The regex to filter out invalid UTF-8 forms, for strict adherance to UTF-8, is as follows perl -l -ne '/ ^( ([\x00-\x7F]) # 1-byte pattern |([\xC2-\xDF][\x80-\xBF]) # 2-byte pattern ...


5

I have reformulated your questions a bit, for reasons that should appear evident when you read them in sequence. 1. Is it possible to config linux filesystem use fixed character encoding to store file names regardless of LANG/LC_ALL environment? No, this is not possible: as you mention in your question, a UNIX file name is just a sequence of bytes; the ...


5

The following Perl command prints the lines that do not contain any Chinese character (Han script). -CIO tells perl that the input and output are encoded in UTF-8. perl -CIO -lne '/\p{Han}/ or print'


5

ß is actually a ligature of ss (in German). Anybody using a a table to convert Unicode or other extended alphabet characters to "safe" characters for things like URLs will likely convert it to ss. Doing this for URLs is quite normal. For example I speak Turkish, where we have letters not found in English such as ö ü ı â ğ ç ş İ. These characters are not ...


5

This behavior is caused by a known bug in poppler (reported at least here and here) related to unicode characters. There is no fix at the moment, you just have to follow the bug tracker.


5

After snooping around various forums and some Stack-overflow post I found the issue. Here is an explanation to better understand and 'see' the root cause of the errors. First off if you are writing code on Windows, better arm yourself with Notepad++. It's basically the Swiss Army knife of text-editors in my opinion. Now to have a look at the EOL ...


5

Your top example is running with a non-Unicode locale (i.e. ASCII). Check your $LANG environment variable (try export | grep LANG); you will most likely not find a .UTF-8 suffix. Try adding it: export LANG=$LANG.UTF-8 Your other example is running with a UTF-8 locale, which should be the default for recent shells. It seems htop detects your locale and ...


5

The shell interprets the commandline with certain rules which you have to consider here: You can escape shell metacharacters with \ so that it behaves like an ordinary character. You can use single or double quotes and inside these most (with double quotes) or all (with single quotes) shell metacharacters lose their special meaning. Quotes don't have to be ...


4

In bash: #!/bin/bash while IFS= read -r line; do printf "$line" done IFS= is needed to stop IFS stripping occurring, that is, in most cases (where IFS contains whitespace, which is the norm) if you don't make IFS empty, whitespace will be stripped from each side of "${line}" regardless of the fact that you have quoted it. The -r option to read stops ...



Only top voted, non community-wiki answers of a minimum length are eligible