Ubuntu 16

I currently use cat to pipe a logfile from a remote machine to a website log viewer. The logfile has a two-byte signature at the beginning which tells it the file is 16bit or 8bit.

if I nano the file, the file looks like so ...

��2^@0^@1^@8^@0^@4^@2^@8^@ ^@1^@5^@:^@4^@3^@:^@0^@8^@
^@0^@1^@8^@0^@4^@2^@8^@ ^@1^@5^@:^@4^@3^@:^@0^@8^@

If I cat filename I can still see the first 2 bytes at the beginning of the file

▒▒20180428 15:43:08
20180428 15:43:08

If I do iconv -f utf-16 -t utf-8 filename It looks great in shell ...

20180430 03:57:46
20180430 03:57:46

but shows up on the website with the first line partially encoded

20180430 03:57:46 ㋾ ㄀㠀 㐀㌀   ㌀㨀㔀㜀㨀㐀㘀 嘀攀爀猀椀漀渀           㴀 㐀⸀ ⸀ ਀

Now if I reverse the log iconv -f utf-16 -t utf-8 filename | tac it shows up on the website perfectly.

Not sure how to proceed.

  • 1
    Your comment worked well with cat filename | tail -c +3 and in reverse cat filename | tail -c +3 | tac . – Curious Sam Apr 30 '18 at 9:17
  • I tried to upvote your comment, then I downvoted by mistake. Sorry. – Curious Sam Apr 30 '18 at 9:18
  • Could you post a hexdump of the beginning of the file before and after your iconv conversion? – hmakholm left over Monica Apr 30 '18 at 11:19
  • @dsstorefile1 Please post this as an answer – Michael Mrozek Apr 30 '18 at 16:57

If you want to remove the first 2 bytes, you could try:

dd if=orginal_file of=new_file ibs=2 skip=1

That will output new_file with the first 2 bytes skipped.

| improve this answer | |

If you're only trying to remove the first two bytes you can do so with tail -c +3. For instance:

$ printf 'hello\n' >> file1
$ tail -c +3 file1
$ _
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.