0

In my Linux env, for some unknown! reason when I process a raw unformatted file which I copied from Win7prof (was encoded in utf-16), the resulting file is all jumbled. It will print OK in Linux using cat command, but if I vi, its also jumbled like this:

Username;Full Name;Groupsï¼å¾çæçæ¸ææ´æã¬äçæ°æ°âä¸ææ´æã¬ççæçççà¨æâ¸ææ´æ¤çâ¸çæçæççæçäæ¼çâ¸çæçæççæçäæ¼çæææ°æâ¸ææ¼æ´ã¬ääåå¼ççæççà¨æâ¸ææ´æ¤çâ¸çæçæççæçäæ¼çæææçäæ¼çæææ°æâ¸ææ¼æ´ã¬æ¨æ¤çæâ´ççæççà¨ã¬^@æææ´æ¤çâ¸æææ¤æäæ¼çæææ°æâ¸ææ¼æ´ã¬äææ´æ¤çææ´æ¤çâ¸æææ¤æäæ¼çæææ°æâ¸ææ¼æ´ã¬æ¨æ¤çæâ´ççæççà¨ã¬^@ææçæ¼æ¸â¸ææ°æçææ æçäæ¼çæææçææ æçäæ¼çæææ°æâ¸ææ¼æ´à¨ã¬^@ææçæ¼æ¸â¸ææ°æçææ æçäæ¼çæææ°æâ¸ææ¼æ´ã¬æ¨æ¤çæâ´ççæ¼æ´ã¬ä¸äå¼ççæççà¨ã¬^@ææçæ¼æ¸â¸æ


the same file when moved to Windows, opens up with first few words in English, then the rest of the file prints in Chinese characters!
looks like:

Username;Full Name;Groups＊嗾猀攀爀渀愀洀攀㬀䘀甀氀氀 一愀洀攀㬀猀琀愀琀甀猀਀愀⸀愀洀椀琀⸀猀愀爀愀猀眀愀琀䀀漀爀愀挀氀攀⸀挀漀洀㬀䄀 匀愀爀愀猀眀愀琀愀⸀愀洀椀琀⸀猀愀爀愀猀眀愀琀䀀漀爀愀挀氀攀⸀挀漀洀㬀䔀䄀匀开甀猀攀爀猀਀愀⸀愀洀椀琀⸀猀愀爀


When I check the char-set of the file, it returns:
application/octet-stream
and I can't run iconv command because it rejects octet-stream
as a character set!

The only sure thing I know is that file formats between my linux and Windows have been compatible all along, I've modified files on linux, ftp back to windows for ever without any issues until now with this command:

[me@localhost fkhodada]# awk -F';' -v OFS=';' '

{ for (i=3; i<=NF; i++) print $1,$2,$i }
' file1 > final

4
  • Windows does really weird things with encodings. But do make sure the output of locale(1) on the linux box is correct.
    – o11c
    Nov 11, 2014 at 5:27
  • Please let us know: ------ 1. How was the file created? (application, possibly procedure) ------ 2. How was the file transferred to Linux? Could not there be any conversion (like the implicit one in FTP)? ------ 3. Please post a hex dump of the file: hd file or xxd -g1 file or od -tx1c file. ------ 4. What you wrote: "the same file when moved to Windows, opens up with first few words in English, then the rest of the file prints in Chinese characters!" strongly suggests that the file content is corrupted. Why do not you consider this possibility?
    – pabouk - Ukraine stay strong
    Nov 11, 2014 at 7:47
  • Addition to 4. ...or opening in a correct application is fine? ------ 5. If yes, what is the correct application and what is the application which shows the Chinese characters?
    – pabouk - Ukraine stay strong
    Nov 11, 2014 at 7:50
  • You my friend asked all the right questions, and that lead me to solve the mystery. I started fresh with a small txt file, just to go through and replicate all the steps I was about to report back to you. In that process I realized that the problem lies in default settings of notepad in windows. I'll explain in more detail tomorrow when I'm fresh.
    – user2793091
    Nov 14, 2014 at 2:25

1 Answer 1

Reset to default
0

After playing with this scenario I realized the encoding problem was in the original file I exported from my windows env.
So I did repeated the same routine, this time, at "save as" argument, chose encoding type to UTF8 and the file looked fine in Linux, and after modified and exported back to windows, looked fine in Windows notepad too.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .