3

We have a server (camera) sending RTSP video packets via UDP. At a customer site it travels over several hops, one of which may be an unreliable WiFi link that drops the odd packet or five. Usually this goes unnoticed but sometimes it kills the stream for some seconds and causes customer displeasure (I know, their cr*p network is somehow our problem...)

On testing using tc to simulate a dodgy connection we have found an odd situation: If we break the connection in the return direction (packets silently discarded), after some seconds the flow of UDP packets from our camera stops, even though the RTSP client (Live555 Wis-Streamer) still believes it's merrily squirting UDP packets up the pipe.

This is odd, as obviously UDP packets are not ACK'd and the physical link never drops, so our system has no way of knowing that the packets are dropping into the bit bucket further up the chain and the streamer has no way of knowing that no-one is listening to it (the streamer session timeout does not expire until later).

EDIT: We see ARPing (Who has <client>) at the moment the UDP packets stop coming but none prior to that which would tell the stack the connectivity has dropped.

So I have two questions:

  1. Is there some other mechanism by which the networking stack can tell the connection has issues?
  2. Does the network stack silently drop packets under certain circumstances?

To demonstrate our testing setup:

Normal state, connectivity both ways:

Our server <==> Switch <==> TC  <==> Switch <==> PC
                 |                    |
  Wireshark <-- TAP                   |
                                      |
  Wireshark <----------------------- TAP

Fault state, TC dropping packets going back to our server:

Our server --> Switch --> TC  <==> Switch <==> PC
                 |                    |
  Wireshark <-- TAP                   |
                                      |
  Wireshark <----------------------- TAP
  • Who is sending the "Who has <client>" ARP requests? And what are the criteria used by tc to identify the packets that are dropped? – Mark Plotnick Jun 3 '15 at 19:33
  • The server starts sending Who has <pc>? requests coincident with stopping sending UDP packets. – John U Jun 4 '15 at 9:12
  • tc is set to just drop everything in one direction (server to pc) – John U Jun 4 '15 at 9:14
  • So tc is not dropping the server's Who has <client> ARP requests, and the client is seeing those requests and the server is getting the replies? – Mark Plotnick Jun 4 '15 at 13:56
2

Once you break the connection, you should start receiving ICMP destination unreachable packets to notify you that the connection is broken. This is normal IP behavior.

There are tools which will monitor for and display ICMP packets. Other tools can be used to dump or capture all traffic matching a selection criteria.

It may be better to use a custom proxy server to drop your packets.

  • 1
    Does it depend on the nature of the break (drop vs reject the packets)? We're not seeing anything when we break the connection, only that after some random-ish time our server stops producing UDP packets. Either way, I wonder if Wireshark or the switch is somehow concealing those messages from us? I'd very much hope not as that's not Wireshark's way and the switches have pretty decent port-mirroring (Netgear ProSAFE). – John U Jun 3 '15 at 13:51
  • 1
    I've double-checked, when we break the connection we are doing so in software by just dropping packets into the bit-bucket silently so there is no "comeback" from this in the form of rejections or error messages. – John U Jun 3 '15 at 15:25
  • 1
    @JohnU If you are accepting and dropping the packets, then I would be concerned you get the same behavior when you aren't dropping packets. Depending on your filter Wireshark may not be catching the ICMP packets. – BillThor Jun 4 '15 at 0:42
0

Well it looks like it was the ARP table going stale (even though we're streaming UDP like crazy, that doesn't kick the ARP timeouts, and TCP traffic is much more sparse under normal operation) upping the timeouts stopped the issue from appearing for "breaks" of less than ~2mins (at which point the RTSP client session times out anyway):

ARP gc_staletime extended from 60sec to 360sec
ARP base_unreachable time extended from 30sec to 240sec

Unfortunately this took a fair bit of poking about as we're on Busybox without an arp command available, but it now seems reliable for the situation we're trying to handle.

I'm still keen to understand how the network stack works - at the moment the ARP table/entry goes stale it stops sending out packets yet seemingly doesn't cause errors further up the chain in the code that's trying to send packets.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.