No packets are received for several seconds and then ~6 are sent back in quick succession.
This is symptomatic of two similar phenomena: network congestion or network discards (usually due to congestion).
In the first case, a router between here and there has a burst of traffic unrelated to your activities which cause your traffic to be buffered in some intermediate router. They'll wait their turn until there is bandwidth open to send them on their way. Congestion like this could result from anything from a sudden spike in YouTube traffic (new kitten video!!!) or even something like an attempted SYN_ACK attack. In practice, there are far more attempted malfeasant attacks than we'd like because there are a huge number of infected machines out there which will spontaneously throw traffic at a random device somewhere on the planet. Even though SYN_ACK and similar attacks are now quashed shortly after detection, even the detection and quashing can keep a router busy for a few seconds.
The second case is that your traffic hits an overloaded device and it does not buffer the traffic. Either because it has no extra buffer memory, or because buffering often causes its own problems. For example, "I've buffered traffic because the router one hop over was too busy right now so as soon as it becomes available, I'll hit it with my stored traffic, thus making it over-busy…" ad infinitum. In this case, your TCP connection will begin its exponential backoff which will cause a delay on your (sending side). Historically this was a splendid approach to coping with the very bursty internet. There are a big handful of problems with this core part of the transmission protocol but no great solutions.
Unfortunately without the devoted assistance of your ISP, telco, and various system administrators, such lag spikes are nearly impossible to diagnose. In all likelihood, the device that is oversubscribed for its peak traffic is located somewhere completely inaccessible to you and its operator may not even know it is overloaded nor care.
The Internet Protocols were designed for best effort delivery giving no guarantees that a packet would ever make it to its destination. That it works as well as it does under loads that were never imagined is, to me, a minor miracle. If you need better than the public internet can provide, someone would probably be happy to sell you a dedicated line from you to your destination for some arbitrarily high price. Otherwise, like freeway traffic or random overlong queues at the grocery store, it might just have to be an inconvenience of modern life that you just have to live with.
As a side-note, physical proximity is poorly correlated with topological proximity. For a good time, try traceroute destination-host and marvel at just how many devices your traffic traverses between here and there. It is not unusual for a 1km transfer to go a megameter and 20 devices to get to its destination.
added in response to comment:
I have never noticed the issue occurring when I use PuTTY on a windows partition on the same machine.
Does your statement "on a windows partition" mean "running on windows"? I'll assume it does.
Without more precise data I would first presume that your not noticing it was most likely you not noticing it but I'm not a certain of that. An alternate hypothesis is that the latency spikes are not happening with PuTTY which apparently does use a different SSH implementation. If you could quantify the lack of latency spikes as you did in the ping graph above that would help distinguish between network and client issues.
To get more transfer data, I'd use PuTTY scp to copy large files between your machine and the host in question. You can use wireshark to record inter-packet times.
There are a couple of flaws with the ping test in your graph. The first is that ping uses a ICMP packets which are quite distinct from TCP/IP and are frequently given less priority than IP traffic and are more likely to be discarded by intermediate routers. As a quick check, those data are useful, but if you want to track TCP/IP connections it is best to use IP packets which is why I recommend scp. You could also use the same scp / wireshark combination under unix for comparative purposes.
The other problem with the ping test is that 60 seconds is too short a period to get a decent picture of periodic behavior. Since you already appear to have summarization tools at hand, 10 minutes would be better than 1 minute and an hour better still.
When testing, I'd vary the data I'm passing between machines. Here's a very quick-and-dirty script to generate files with much entropy and almost none:
#!/usr/bin/env python2.7
import random
def data_bytes(outf, ordered=False):
"""write a series of ordered or random octets to outf"""
for block in range(1024):
for char in range(1024):
if ordered:
c = char % 0x100
else:
c = random.randint(0, 0xff)
outf.write(chr(c))
def main():
with open('random.dat', 'wb') as outf:
data_bytes(outf, ordered=False)
with open('sequen.dat', 'wb') as outf:
data_bytes(outf, ordered=True)
if __name__ == '__main__':
main()
please forgive me if this bit is patently obvious.
Your anecdotal observation makes this an interesting question. It does need hard data to get further.
