4

I have a 300 lines file with ^@ character between each characters in the file.

(I cannot post the entire contents for security reasons, so I am pasting only the first line)

[mercury@app01 ftp_logs]$ cat cl.txt
2015-01-22 03:00:01; local;

Now, when I vi the file, I see the same contents as following:

2^@0^@1^@5^@-^@0^@1^@-^@2^@2^@ ^@0^@3^@:^@0^@0^@:^@0^@1^@;^@ ^@l^@o^@c^@a^@l^@;^@

Since cat wasn't displaying the ^@ characters, naturally I thought grepping for a certain string would work in cat, but surprisingly, it wasn't.

[mercury@app01 ftp_logs]$ cat cl.txt
2015-01-22 03:00:01; local;
[mercury@app01 ftp_logs]$ cat cl.txt | grep local
[mercury@app01 ftp_logs]$

After replacing the null bytes with sed, file is now readable in vi and grep returns the result from cat.

[mercury@app01 ftp_logs]$ sed -i 's/\x0//g' cl.txt
[mercury@app01 ftp_logs]$ cat cl.txt | grep local
2015-01-22 03:00:01; local;
[mercury@app01 ftp_logs]

Questions:

1) Why didn't grep work before replacing the null bytes, given that null bytes weren't getting displayed. Does it mean grep saw the ^@ characters even though it wasn't displayed in the terminal?

2) This makes me wonder whether it is recommended to use cat -v or vi to read files on production servers since catseem to be good in hiding stuff?

3) The file in question is an auto-generated file from a Windows machine. Under what circumstances does the ^@ find its way into a file.

5

The format of the file is probably little-endian UTF-16. Some apps on Windows seem to default to this, and it causes a lot of porability problems.

vi represents ASCII-Nul (numerically zero) valued bytes as '^@' (control-At). You can actually enter zero-valued bytes in vim with the control-shift-@ chord.

grep must see the ACII-Nul bytes, rather than interpret the file as UTF-16, and then seeing the Unicode code points for '2' or '0' or whatever. I don't see an option in the GNU grep man page for making it deal with UTF-anything.

cat doesn't show the ASCII-Nul btyes, the terminal emulator in question would show them, but whatever terminal emulator you're using is ignoring them. If you use cat cl.txt | od -x or better, cat cl.txt | xxd, you'll see the ASCII-Nul bytes in the output of cat. If you see something like 'ffef' or 'efff' as the first two bytes of the file, those are the "byte order mark" promulgated by Microsoft against all common sense.

I'm not sure what to recommend to transliterate UTF-16 to ASCII or UTF-8, iconv maybe, but I've never used it.

  • So, basically cat's inability to show the ASCII-Nul bytes got me confused. – Sree Feb 9 '15 at 7:03
  • 1
    @sree - cat just writes bytes (not characters!) to stdout. The way you invoked cat, stdout was the TTY of the terminal emulator. That TTY or the terminal emulator chose not to show the ASCII-Nul bytes and confused you. – Bruce Ediger Feb 9 '15 at 14:02
4
  1. Yes, grep sw the ^@ characters. cat is printing the characters to the terminal, but they are characters you can't see. Just because you can't see the characters doesn't mean they aren't there.

  2. Your choice/preference, depending on which one works best for what you need. Remember, though, that vi has the possibility to change the file.

  3. ^@ is not a natural character. The Windows program is actively putting those characters there. To find out why, you'd have to ask the programmer. Most likely, the Windows program is assuming that characters are 16 bits wide, and your Unix machine is assuming that characters are 8 bits wide.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.