Tag Info

Hot answers tagged

23

As noted by others, there isn't really an answer to this: filenames and paths do not have an encoding; the OS only deals with sequence of bytes. Individual applications may choose to interpret them as being encoded in some way, but this varies. Specifically, Glib (used by Gtk+ apps) assumes that all file names are UTF-8 encoded, regardless of the user's ...


21

When Vim reads an existing file, it tries to detect the file encoding. When writing out the file, Vim uses the file encoding that it detected (except when you tell it differently). So a file detected as UTF-8 is written as UTF-8, a file detected as Latin-1 is written as Latin-1, and so on. By default, the detection process is crude. Every file that you open ...


17

The char type in C is one byte, but it's intended for ASCII characters; there are variable-width encodings like UTF-8 that can take up many bytes per character. wc uses the mbrtowc(3) function to decode multibyte sequences, depending on the locale set by the LC_CTYPE environment variable. If you set the locale properly, you should get the same result for all ...


15

I guess you see this � invalid character because the name contains a byte sequence that isn't valid UTF-8. File names on typical unix filesystems (including yours) are byte strings, and it's up to applications to decide on what encoding to use. Nowadays, there is a trend to use UTF-8, but it's not universal, especially in locales that could never live with ...


15

You're right about the line endings being important; both OSes expect the line to end with "\n", but Windows also adds a "\r" before that that unix doesn't expect, so unix programs will output the "\r" in their own way. The file doesn't actually end with the two characters "^" and "M", that's just a common way to represent unprintable characters. Programs ...


14

It should be pointed out that Mac OS X uses \n a.k.a linefeed (0x0A) now, just like all other *nix systems. Only Mac OS versions 9 and older used \r (CR). Reference: Wikipedia on newlines.


13

Edit: I've fixed a typo-bug in the regex.. It needed a '\x80` not \80. The regex to filter out invalid UTF-8 forms, for strict adherance to UTF-8, is as follows perl -l -ne '/ ^( ([\x00-\x7F]) # 1-byte pattern |([\xC2-\xDF][\x80-\xBF]) # 2-byte pattern ...


13

I think you probably want iconv. It's for converting between codesets and supports an absurd number of formats. For example, to strip anything not valid in UTF-8 you could use: iconv -c -t UTF-8 < input.txt > output.txt Without the -c option it'll report problems in converting to stderr, so with process direction could you save a list of these. ...


13

At a guess, Your locale uses UTF-8 encoding, and About 10% of your file consists of characters which require more than one octet to encode into UTF-8. By the way, from man wc: -c, --bytes print the byte counts -m, --chars print the character counts


13

It is known as carriage return. If you're using vim you can enter insert mode and type CTRL-v CTRL-m. That ^M is the keyboard equivalent to \r. Inserting 0x0D in a hex editor will do the task. How to remove? You can remove it using the command perl -p -i -e "s/\r//g" filename. As the OP suggested in the comments of this answer here, you can even try a ...


12

Running the locale command will give you information about your locale settings; the character encoding is given by the LC_CTYPE setting. Under Ubuntu, the default locale settings are given in /etc/default/locale. You can change the character encoding by setting LC_CTYPE in your ~/.profile on the VPS, e.g. export LC_CTYPE=en_US.UTF-8 You'll have to make ...


12

Short answer: restrictions imposed in Unix/Linux/BSD kernel, namei() function. Encoding takes place in user level programs like xterm, firefox or ls. I think you're starting from incorrect premises. A file name in Unix is a string of bytes with arbitrary values. A few values, 0x0 (ASCII Nul) and 0x2f (ASCII '/') are just not allowed, not as part of a ...


12

Assuming that "foreign" means "not an ASCII character", then you can use find with a pattern to find all files not having printable ASCII characters in their names: LC_ALL=C find . -name '*[! -~]*' (The space is the first printable character listed on http://www.asciitable.com/, ~ is the last.) The hint for LC_ALL=C is required (actually, LC_CTYPE=C and ...


11

Run reset. From the man page: When invoked as reset, tset sets cooked and echo modes, turns off cbreak and raw modes, turns on newline translation and resets any unset special characters to their default values before doing the terminal initialization described above. This is useful after a program dies leaving a terminal in an abnormal state.


11

Change the character translation in PuTTY to UTF-8.


9

tput sgr0, tput rmacs, or tput reset. reset is usually but not always even more complete than tput reset. (Turn off all special output modes; turn off alternate character set, which is usually but not always included in sgr0; send terminal reset string which often does those and more.) It can be useful to embed tput sgr0 at the beginning of $PS1.


9

Your locale information isn't set up correctly on the server machine. Specifically, the LC_CTYPE variable, which indicates the encoding of characters on the terminal, isn't set correctly (or, I suspect, at all). In your local terminal, run locale to see your locale settings. You'll probably see (amongst other lines) something like LC_CTYPE=en_US.UTF-8; what ...


9

The thing is, the kernel doesn't care one bit how the applications interpret the data it is given as a filename. Let's imagine I have a C application that deals with exclusively UTF-16 strings. And I enter, via a properly configured input method, the ∯ symbol (Unicode 0x222F) into the "Save As" prompt/dialog. If the application doesn't do any form of ...


8

With bash, zsh, GNU echo or some implementations of ksh on some systems, this can be decoded simply by echo -e after replacing all % with \x. url_encoded_string="%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" temp_string=${url_encoded_string//%/\\x} printf '%s\n' "$temp_string" # output: \xD1\x80\xD0\xB5\xD1\x81\xD1\x83\xD1\x80\xD1\x81\xD1\x8B echo -e ...


8

From this wikipedia article, FF FE means UTF16LE. So you should tell iconv to convert from UTF16LE to UTF8: iconv -f UTF-16LE -t UTF-8 dotan.csv > fixed.txt


7

· in latin1 is the byte sequence c2 b7. That same byte sequence interpreted in UTF-8 is the character ·. So what's happening is that Emacs believes your terminal displays UTF-8 (Unicode) but your terminal actually displays latin1. PuTTY supports UTF-8, so the easiest solution is to tell it to use UTF-8, at least when connecting to this host. As an added ...


7

It isn't always possible to find out for sure what the encoding of a text file is. For example, the byte sequence \303\275 (c3 bd in hexadecimal) could be ý in UTF-8, or ý in latin1, or Ă˝ in latin2, or 羸 in BIG-5, and so on. Some encodings have invalid byte sequences, so it's possible to rule them out for sure. This is true in particular of UTF-8; most ...


7

You can do: man ascii to see the whole set of ascii characters, or you can just run the command ascii. $ ascii Usage: ascii [-dxohv] [-t] [char-alias...] -t = one-line output -d = Decimal table -o = octal table -x = hex table -h = This help screen -v = version information Prints all aliases of an ASCII character. Args may be chars, C \-escapes, ...


6

By default, iconv refuses to convert the file if it contains characters that do not exist in the target character set. Use //TRANSLIT to “downgrade” such characters. iconv -f utf-8 -t iso8859-1//TRANSLIT


6

the file command makes "best-guesses" about the encoding. Use the -i parameter to force file to print information about the encoding. I created two files containing german umlauts. one encoded in utf-8 and one encoded in iso-8859-1. $ file -i * file1: text/plain; charset=utf-8 file2: text/plain; charset=iso-8859-1


6

Why not just: # grep -e "[a-zA-Z0-9]\|^$" file.txt


6

Try with printf: for((i=32;i<=127;i++)) do printf \\$(printf '%03o\t' "$i"); done;printf "\n" See also : BASH FAQ


6

It is a double byte format, probably UTF-16. See if you can identify a BOM (Byter Order Mark) header at the beginning of the file. This will tell you the encoding if it is present, although it may not be. Note that a text editor may hide this from you, so you will probably need to look at the file with a hex dump utility such as od or something similar to ...


6

For external representations, UTF-8 is definitely the standard. Some 8-bit encodings are still strong (mostly in Europe) and some 16-bit encodings are still strong (mostly in East Asia), but they are clearly legacy encodings, on their slow way out. UTF-8 is standard not only on unix, but also on the web. For internal representations, there's no such ...


6

You could use the iconv character set conversion GNU utility. Here's an example: iconv --from-code=UTF-8 --to-code=MAC INPUTFILE > OUTPUTFILE where INPUTFILEand OUTPUTFILE are the names of your original and converted file (which may be the same). You'd alaso need to substitute UTF-8 by your native encoding, if it's not Unicode, and MAC may not be ...



Only top voted, non community-wiki answers of a minimum length are eligible