How does the processor determine each instruction's opcode and operands: looking into binary file with VIM [closed]

Question

I run the command view /bin/ls on FreeBSD 10.3 and I can see the binary file unmodified:

Then on vim/view I run the command :%!xxd and I can see the file in hex format as follows. I notice at the bottom of the page it is announced by vim that 1708 lines are added and 74 lines are deleted.

I close the vim by :q! command and I open it again with view /bin/ls then I run the vim command :%!xxd -b to see the file in binary format which is shown below. At the bottom of the page it is written 4555 lines are added and 74 lines are deleted.

Now I wonder:

• Why are some lines added and some deleted when running :%!xxd and :%!xxd -b commands on vim

• On the hex format, i.e. when %!xxd command is run, the line addresses are 00000000, 00000010, 00000020, 00000030 and so on. Looks like that's because each line contains 16 bytes, hence the 0x10 increment makes sense.

• On the binary format, i.e. when %!xxd -b command is run, the line addresses are 00000000, 00000006, 0000000c, 00000012 and so on. Looks like that's because each line contains 6 bytes, hence the 0x06 increment makes sense.

• Previously I was believing that each binary file contains each processor instruction in a single line and at the beginning of each line there is the relative address of that instruction starting from 0 for the first instruction. But Based on my observation of binary files on vim, it is not true. Now i wonder how the processor determines each instruction's opcode and operands, if the instructions are not formatted line-by-line on binary files.

---

UPDATE:

The last five lines on hex format are:

00006a70: 0100 0000 3000 0000 0000 0000 4862 0000  ....0.......Hb..
00006a80: 3e03 0000 0000 0000 0000 0000 0100 0000  >...............
00006a90: 0100 0000 0100 0000 0300 0000 0000 0000  ................
00006aa0: 0000 0000 8665 0000 d500 0000 0000 0000  .....e..........
00006ab0: 0000 0000 0100 0000 0000 0000 0a         .............

The last five lines on binary format are:

00006aa4: 10000110 01100101 00000000 00000000 11010101 00000000  .e....
00006aaa: 00000000 00000000 00000000 00000000 00000000 00000000  ......
00006ab0: 00000000 00000000 00000000 00000000 00000001 00000000  ......
00006ab6: 00000000 00000000 00000000 00000000 00000000 00000000  ......
00006abc: 00001010                                               .

Therefore, I thinks the total number of bytes are the same for both hex and binary formats, I mean the address of the last byte of code for both would be 0x6abc.

Answer 1

Why are some lines added and some deleted when running :%!xxd and :%!xxd -b commands on vim

Because vim counts 0x0a as a newline, and the binary file contains those (74 in your version of ls...), so when swapping between the raw binary and some other form, those 74 "lines" in the binary will be removed, and new lines added for the (much more verbose) hex display. vim is just counting the 0x0a it sees.

Now i wonder how the processor determines each instruction's opcode and operands

Magic! It's complicated, and there are many books on this topic. Briefly, the linker (or equivalent) for a particular binary format (ELF in your case though there do exist other formats—a.out, Mach-O, ...) will indicate a start address

$ readelf -h /bin/ls
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x37f0
...

that after the program is hoisted into memory execution of the opcodes will begin at. The start address is usually (but may not be) located somewhere in the .text section of the binary:

$ objdump -DS /bin/ls | less -p .text
...

Which on my OpenBSD system shows:

Disassembly of section .text:

00000000000037f0 <revnamecmp-0x460>:
    37f0:       49 89 e4                mov    %rsp,%r12
    37f3:       48 83 ec 08             sub    $0x8,%rsp
    37f7:       48 83 e4 f0             and    $0xfffffffffffffff0,%rsp
    37fb:       48 83 c4 08             add    $0x8,%rsp
...

Books perhaps worth a peek at include "Assembly Language Step-by-Step" by Jeff Duntemann and for ELF "Learning Linux Binary Analysis" by Ryan O'Neill.