Why does traceroute display many ip addresses for the same hop?

Question

Here is the route path from my home to sina.com.cn.

 traceroute -n   sina.com.cn
traceroute to sina.com.cn (202.108.33.60), 30 hops max, 60 byte packets
 1  192.168.31.1  0.476 ms  0.587 ms  0.695 ms
 2  140.0.5.1  2.557 ms  2.699 ms  3.065 ms
 3  221.11.155.65  4.501 ms * 221.11.165.9  5.045 ms
 4  * 221.11.156.18  26.480 ms 221.11.165.233  22.950 ms
 5  219.158.9.97  14.176 ms * 219.158.19.149  21.472 ms
 6  219.158.9.97  18.142 ms 219.158.8.81  44.856 ms  52.539 ms
 7  124.65.194.190  53.162 ms 219.158.8.81  50.614 ms 124.65.194.190  47.266 ms
 8  124.65.194.190  50.760 ms 61.148.143.26  49.351 ms  53.515 ms
 9  210.74.176.138  43.056 ms  43.286 ms 61.148.143.26  53.712 ms
10  202.108.33.60  46.385 ms 210.74.176.138  42.896 ms  46.931 ms

192.168.31.1 is my home router.
140.0.5.1 is my public ip the ISP provides.

curl  ifconfig.me
140.0.5.1 

In the third line ,it says

3  221.11.155.65  4.501 ms * 221.11.165.9  5.045 ms

Why there are two ip addresses 221.11.155.65 and 221.11.165 ? What does it mean?

Does the packet jump from 140.0.5.1 to 221.11.155.65 ,then jumps from 221.11.155.65 to 221.11.165 ?

Answer 1

What traceroute give you here as information is that you have at least 2 routes to reach your destination starting from router:

           IP adresses                 Internet operator

           140.0.5.1                   Firstmedia - Indonesia
             /    \
            /      \
221.11.155.65      221.11.165.9        China Unicom - China

This is a normal situation for a huge Internet operator: to have many routes to resist to normal incidents on the Internet.

The FreeBSD -a (-A on Linux) option of traceroute may help you identify to which operator every IP address belongs.

Here is an example of a larger tree of routes you can get even from your mobile phone through a network tool like server http://ping.eu/traceroute/ :

Answer 2

From the traceroute(8) manual on OpenBSD:

Three probes (the exact number can be changed using the -q option) are sent and a line is printed showing the TTL or hop limit, address of the gateway, and round trip time of each probe. If the probe answers come from different gateways, the address of each responding system will be printed.

The Linux manual will have similar wording.

The multiple IP addresses that you see are the gateways responding to the individual probes at specific hop limits.

In your case, the three probes resulted in replies that, at hop limit 3, came back to you from the gateways at 221.11.155.65 and at 221.11.165.9.

So, the answer is: No, the packet does not jump between the two hosts listed on that line, there are three probes sent and they take two different routes from 140.0.5.1.

Answer 3

What traceroute does is use the option fields as it sends internet control message protocol (icmp) packets. Each Gateway or routing point in the network reads packets on one interface, decides where they ought to go, and writes them out on another interface. That interface is presumably closer to the destination. While the router is forwarding the packet it also makes modifications to the packet header. It reduces the "time to live" or hop count field by one. Each Gateway in the path of the packet toward the destination decreases this field.

When the hop count field drops to zero, many routers will send back an icmp message saying where the packet was dropped. In order to find which IP address is a specific distance away, traceroute will send packets with the options of time to live & give me a response when it times out. messages with hop counts starting at 1 and ramping way up will each return the IP address of progressively more distant routers. Because traceroute does this multiple times you're going to get back (if you have a richly connected network, as is the internet) multiple answers at some of the counts. it may be the case that a particular Gateway will answer at different amounts away because the route to that gateway went through different hops.