I am facing an issue in my project.We support an App which is hosted on a unix server.

Partners try to access the same from many regions. Now some users in an region say "R" are complaining about slowness and we have reason to believe that it might be due to a network issue which is at their end.

What are some commands I can run in their system in their terminal through which I can prove to them that it is an network issue at their end?

Also some commands which can prove to them that other web applications in the past few minutes have also taken a lot of time from their systems?

I am very new to UNIX.Thanks in advance.

  • What data would you consider would "prove" to them that the problem is at their end? – a CVn Jul 19 '16 at 20:43

Inherent limitations

The issue could be variable (e.g. congested link to ISP or congestion within ISP). It could also be horrible ("firewall" or even "anti-virus" doing deep packet inspection); the tools below might not show any problem at all. They're worth having, but there is a limit to how much you can achieve just typing commands into a terminal.

2 tests you should know

  1. Use ping to measure round-trip latency to servers over ICMP/IP. You can also traceroute or tracepath to your server, and check how much round-trip latency there is to the first few hops. You're mainly trying to check for symptoms of bufferbloat, so be aware that only happens when the link is fully used! ("latency under load" measure).

  2. You can check available web download bandwidth (single-stream) just using wget or curl --remote-name to download a file. If you're uninspired I suggest downloading a Linux :-). Find the download link and use "copy link location" from the right-click menu. You probably don't have to let the download run to completion because it'll show the current download rate - use Control+C to cancel it. You could test a mirror in the same region as your server (this is potentially significant). I guess if you're considering the terminal, it's good to know wget exists. Personally I'd prefer to use http://testmy.net/mirror.

That's basically it, from the information you've given. There's a caveat with one of the results from ping, which I've highlighted below.

ping is excellent for initial testing. traceroute is an expert tool. I only suggest traceroute as a way to try and illustrate bufferbloat, if that's what ping seems to show... it may actually be better to use ping on the routers you see in traceroute.

Low download rates as a direct cause can easily be over-estimated. Webapps don't need to serve much data to respond to user requests unless there are uncached images. E.g. unix.stackexchange.com is 75K and takes 0.2s to download at 4Mb/s. But it's easy to run a test, and provides a little data point to fit into the puzzle.

How much packet (ping) loss is too much?

Any noticeable packet loss rate can limit download rate, and particularly over trans-continental distances.

Unfortunately the effect of loss on short transactions is a bit more complicated than that. It looks like a single loss probably won't cause more than 100% increase for transfers around ~20Kb. Unless the first packet from the server (or client) is dropped, in which case it won't recover until a full "receive time out" - 3 seconds.

There's an issue/caveat when measuring loss, in that it could be affected by packet size. When measuring loss with ping, you should notice that it uses small packets by default. This is similar to the first packets from client and server (SYN / SYN-ACK respectively). Putting this together, if you see 5% loss when running ping $SERVER without options, you wouldn't expect a perfect experience using that web app. (I.e. out of 20 user actions, expect 1 of them to take 3 seconds before anything happens at all. It won't be mitigated by persistent connections given common web server configurations)

You can check statistics for full-size packets e.g. ping -s 1400 on unix. In principle there could be yet more factors ("prioritization" at the router aka QoS) and what you specifically want are TCP retransmit details from the specific application, gathered from the kernel or a packet trace.

Note that from an endpoint, it's very difficult to distinguish whether a link is congested, v.s. whether the link is physically unreliable. Packet loss is how routers tell TCP to slow down; more congested links will have more packet loss. I think the best you could expect is to identify ("prove") a link with high packet loss, and ask someone with access for investigation or monitoring.

  • +1. mtr is also a nice constantly-refreshing traceroute. It has a GUI (GTK) interface, but I prefer the ncurses text-only version (mtr-tiny package on debian). – cas Jul 20 '16 at 5:19
  • Wow!! Thank you so much.. Did not expect such a detailed answer and that too so soon. Thanks once again. – user1741642 Jul 20 '16 at 19:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.