-1

I run the command view /bin/ls on FreeBSD 10.3 and I can see the binary file unmodified:

enter image description here

Then on vim/view I run the command :%!xxd and I can see the file in hex format as follows. I notice at the bottom of the page it is announced by vim that 1708 lines are added and 74 lines are deleted.

enter image description here

I close the vim by :q! command and I open it again with view /bin/ls then I run the vim command :%!xxd -b to see the file in binary format which is shown below. At the bottom of the page it is written 4555 lines are added and 74 lines are deleted.

enter image description here

Now I wonder:

  • Why are some lines added and some deleted when running :%!xxd and :%!xxd -b commands on vim

  • On the hex format, i.e. when %!xxd command is run, the line addresses are 00000000, 00000010, 00000020, 00000030 and so on. Looks like that's because each line contains 16 bytes, hence the 0x10 increment makes sense.

  • On the binary format, i.e. when %!xxd -b command is run, the line addresses are 00000000, 00000006, 0000000c, 00000012 and so on. Looks like that's because each line contains 6 bytes, hence the 0x06 increment makes sense.

  • Previously I was believing that each binary file contains each processor instruction in a single line and at the beginning of each line there is the relative address of that instruction starting from 0 for the first instruction. But Based on my observation of binary files on vim, it is not true. Now i wonder how the processor determines each instruction's opcode and operands, if the instructions are not formatted line-by-line on binary files.


UPDATE:

The last five lines on hex format are:

00006a70: 0100 0000 3000 0000 0000 0000 4862 0000  ....0.......Hb..
00006a80: 3e03 0000 0000 0000 0000 0000 0100 0000  >...............
00006a90: 0100 0000 0100 0000 0300 0000 0000 0000  ................
00006aa0: 0000 0000 8665 0000 d500 0000 0000 0000  .....e..........
00006ab0: 0000 0000 0100 0000 0000 0000 0a         .............

The last five lines on binary format are:

00006aa4: 10000110 01100101 00000000 00000000 11010101 00000000  .e....
00006aaa: 00000000 00000000 00000000 00000000 00000000 00000000  ......
00006ab0: 00000000 00000000 00000000 00000000 00000001 00000000  ......
00006ab6: 00000000 00000000 00000000 00000000 00000000 00000000  ......
00006abc: 00001010                                               .

Therefore, I thinks the total number of bytes are the same for both hex and binary formats, I mean the address of the last byte of code for both would be 0x6abc.

closed as too broad by Michael Homer, phk, GAD3R, G-Man, Stephen Kitt Jun 6 '17 at 7:40

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • How many newlines (0x0a) does /bin/ls contain? – thrig Jun 5 '17 at 3:01
  • @thrig I updated the answer, showing the last lines of code – user3405291 Jun 5 '17 at 3:10
  • 1
    :<range>!<command> replaces the lines in the range with the output of the command - hence m lines added and n deleted, where m will probably be the current number of lines and n the previous number. – muru Jun 5 '17 at 3:45
  • What is the question here? – Michael Homer Jun 5 '17 at 3:59
  • @MichaelHomer The question was how does the processor understand which byte is opcode and which byte is operand. Based on format of binary file shown on VIM, there is not a specific address assigned to each processor instruction, so how does the processor know? – user3405291 Jun 5 '17 at 4:29
3

Why are some lines added and some deleted when running :%!xxd and :%!xxd -b commands on vim

Because vim counts 0x0a as a newline, and the binary file contains those (74 in your version of ls...), so when swapping between the raw binary and some other form, those 74 "lines" in the binary will be removed, and new lines added for the (much more verbose) hex display. vim is just counting the 0x0a it sees.

Now i wonder how the processor determines each instruction's opcode and operands

Magic! It's complicated, and there are many books on this topic. Briefly, the linker (or equivalent) for a particular binary format (ELF in your case though there do exist other formats—a.out, Mach-O, ...) will indicate a start address

$ readelf -h /bin/ls
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x37f0
...

that after the program is hoisted into memory execution of the opcodes will begin at. The start address is usually (but may not be) located somewhere in the .text section of the binary:

$ objdump -DS /bin/ls | less -p .text
...

Which on my OpenBSD system shows:

Disassembly of section .text:

00000000000037f0 <revnamecmp-0x460>:
    37f0:       49 89 e4                mov    %rsp,%r12
    37f3:       48 83 ec 08             sub    $0x8,%rsp
    37f7:       48 83 e4 f0             and    $0xfffffffffffffff0,%rsp
    37fb:       48 83 c4 08             add    $0x8,%rsp
...

Books perhaps worth a peek at include "Assembly Language Step-by-Step" by Jeff Duntemann and for ELF "Learning Linux Binary Analysis" by Ryan O'Neill.

  • There is a processor model online, which I'm using to understand how the processor identifies opcode and operands. When I do tick to run processor model, it looks like the instruction decoder will fetch operands from memory, according to the current instruction, and it increments the next instruction address register accordingly. Therefore, looks like identification of opcode and operands is implemented by logic gates configuration. I'm not sure. – user3405291 Jun 5 '17 at 4:39
  • 1
    @user4838962 Please stick to one, well-defined question per post. If you have further questions post them separately. Explain only what you're asking, and keep additional rambling to a minimum. – Satō Katsura Jun 5 '17 at 7:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.