I edited a .txt file in Visual Studio on Windows and then copied that into a HPC server. The file looks fine to me at first, enter image description here

But when I opened it in the linux environment, weird characters showed up (actually, it asks me "sampleID.txt" may be a binary file. See it anyway?). I believe the character encodings are somewhat wrong, but don't know what might have caused this, because when I tried to save this file in visual studio, it told me that "some Unicode characters in this file cannot be saved in the current codepage. Do you want to resave this file as Unicode in order to maintain your data?". Does anyone have an easy way to remedify this file? Thanks a lot! enter image description here

  • Next time please do not paraphrase error and warning messages. I really don't believe it said, "this might be a binary. Do you really wanna see it?". The precise message can be really important when diagnosin an issue.
    – roaima
    Feb 22 '20 at 22:13
  • The "visual studio" you refer to - is that Microsoft Visual Studio on Windows?
    – roaima
    Feb 22 '20 at 22:15
  • Yes, it was Microsoft VS on windows. And for the exact warning message, it was ""sampleID.txt" may be a binary file. See it anyway? ". Feb 22 '20 at 23:10

The first 3 bytes are an incorrectly used byte order mark, converted to utf-8. utf-8 should not use a byte order mark.

The other 3 repeated characters are a (not a -).

These display fine on by terminal, emacs, et al, in Debian Gnu/Linux.

You may need to set your locale properly, to get less to work.

e.g. for UK english (for US change the GBs to US. For others look the up, but ensure that they contain utf8. You should be using utf-8 for all locals these days, other encodings are old and mutually incompatible).

  • Seems like a promising answer. However, could you provide some more general info? For example, where/how do you set the locale; check the locale being used; get the available variables related to the locale settings? i.e. "Here's my output after executing $ locale would clear up some of those.
    – ILMostro_7
    Feb 22 '20 at 22:49
  • You could copy the code to ~/.profile Feb 22 '20 at 22:51
  • 1
    @ILMostro_7 You can read man page of localectl Feb 23 '20 at 9:27
  • FWIW,I was just suggesting an improvement on the answer
    – ILMostro_7
    Feb 23 '20 at 13:41

Your file could be reproduced in a system that use UTF-8 encoding by:

{ printf '\xef\xbb\xbf';
  for i in {3..6}; do 
      printf '%s\r\n' ped1_D$((2*(i-2)+2))_{1..3}_g$i−b1−{s,i}1;
} >file

And then, yes, the command less will ask if the file is binary if the encoding is not UTF-8, which could be reproduced by:

LC_ALL=C less file

And yes, it shows many special characters.

But that only happens in LESS, most other editors: nano, vi, emacs could open the file without being mislead by the DOS encoding.

The easiest way to remove the CARRIAGE RETURN (\r) at the end of the lines and auto remove the un-needed BOM (byte order mark) is to use dos2unix. UTF-8 is a byte oriented format, there is no need to re-order bytes, all bytes work in network order. Only in encodings with 16 or 32 bit characters could the BOM be useful.

dos2unix file

But the real problem in your system is that it is not using the utf-8 default encoding. Most OS now-a-days use utf-8 by default. As could be shown by:

LC_ALL=en_US.UTF-8 less file

Please make sure that locale prints an encoding that has utf-8 in its name, and, if needed, make sure your console is encoding with utf-8: stty -a prints -iutf8.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.