Is there any way to find out, what is the incoming and outgoing bandwidth usage of each hop (for a particular port.)?
Unless you own the network element, or your third-party WAN provider discloses the information, you can only estimate end-to-end available ingress and egress bandwidth along a network path. See below.
Is there exist any tool/utility/application who can serve the purpose.
- For path "available bandwidth estimation" that I mentioned above, you should review Sally Floyd's archive of end-to-end TCP/IP bandwidth estimation tools. I am most familiar with
yaz, which is based on unicast UDP packets.
- To see whether you are dropping packets at any given router hop (which is your bottom-line problem), you can use
mtr; there is also a
win-mtr client, which supports Windows. To see a simple example of how I typically troubleshoot packet drops, see my answer on SuperUser. This technique is most effective at providing visibility to packet drops at the first point where they happen (since
mtr doesn't give much visibility to downstream drops beyond that point until you correct the first).
A simple technique to get a rough estimate of where your drops are is to install
mtr on your server and then run an
mtr session to trace packet loss to a single multicast client while you are transferring your 100M file. For more precise measurements you could use
iperf to saturate the network instead of your 100M file (as long as you coordinate WAN downtime appropriately with other groups in the company).
The rest of my answer is going to use the following diagram for reference:
In the diagram:
- R1 through R5 are IP routers
- S1 and S5 are ethernet switches
- The blue server on 172.16.1.0/24 represents your multicast server.
- C51 through C55 are examples of multicast receivers (could be any number of receivers)
The specifics of the WAN between R1 and R5 usually won't matter much, we just need a baseline topology so we're on the same page.
From what I can tell, you're saying that R1's interface on 172.16.1.0/24 shows about 9Mbps while you are sending the 100MB file and R5's interface on 172.16.5.0/24 shows about 4.2Mbps when clients are receiving via reliable UDP multicast. When you say reliable, I assume that means there is some kind of packet sequencing built into the multicast service, and the client application knows how to request a retransmission from the server.
If this description is correct, there are a few likely causes:
- Link congestion somewhere after R1, as you asserted in your question.
- Performance limitations of any Rx device in the path, including R1 and R5 (such as hitting a multicast replication performance limit)
- You're hitting a throughput limitation of 10M half-duplex ethernet.
Causes 1 or 2 would be revealed by using
mtr. However, cause 3 is worthy of a bit more discussion. 10M/half links provide a maximum of 10Mbps for a unidirectional transfer. If you are sending bi-directional traffic on a 10M/half link, you will typically see substantially less than 10Mbps because of ethernet's CSMA/CD dynamics. On a half-duplex link, ethernet cannot simultaneously transmit and receive; if stations try to do this, their frames will collide, and both stations will delay retransmission for a random time.
I test networks for a living. When I have tested effective bi-directional throughput of 10M/half links, I generally see between 3Mbps and 4Mbps. The numbers you're sharing above sound very similar. I don't have enough evidence to make an accusation, but I would not be surprised if your 10M/half links are the problem; particularly if the link between R5 and S5 is 10M/half.