3

I have a machine that runs a network intensive application that spawns many processes. I noticed recently that the machine is producing ARP requests looking for an IP address that does not exist. I would like to trace down which process on the box is causing the ARP requests to be generated for the sake of troubleshooting (so I can know which part of the application is looking for this non-existant IP).

IP's have been changed but they aren't important anyhow.

I discovered these ARP requests by running tcpdump on another machine on the same network:

# tcpdump -i eth0 arp -t -n
ARP, Request who-has 1.1.1.100 tell 1.1.1.1, length 46

There isn't meant to be a device with the address 1.1.1.100, so I want to find what process on 1.1.1.1 is looking for it.

I tried using ss -np | grep 1.1.1.100 as well as netstat -np | grep 1.1.1.100 (netstat is considered deprecated in place of ss for those curious, ss has most of the same options and is meant to perform the same functions). Neither of these return any results, likely because ss and netstat list open sockets, and the ARP request would pre-date a socket being created.

So how can I discern what process causes an ARP request?

1

ss does show you connections that have not yet been resolved by arp. They are in state SYN-SENT. The problem is that such a state is only held for a few seconds then the connection fails, so you may not see it. You could try rapid polling for it with

while ! ss -p state syn-sent | grep 1.1.1.100; do sleep .1; done

One way to extend the time in this state is to set an arbitrary hard-wired mac address for the IP address in your arp table. Then a connect will take over 30 seconds to timeout, and will be easier to see with ss.

For example, with my eth0 at 192.168.1.1

$ socat tcp:192.168.1.100:80 - 
$ arp -i eth0 -n | grep 192.168.1.100
192.168.1.100 (incomplete) eth0

setting the mac address makes the socat easily visible

$ sudo arp -i eth0 -s 192.168.1.100 80:ef:00:ff:ff:ff
$ socat tcp:192.168.1.100:80 - &
$ ss -p state syn-sent
Netid  Recv-Q Send-Q Local Address:Port Peer Address:Port                
tcp    0      1      192.168.1.1:46608 192.168.1.100:http users:(("socat",pid=20230,fd=3))
  • If the connection is only held for a few seconds would it also be valid to just run a recurring ss? Something like watch -n 1 "ss -np | grep 1.1.1.100"? I like the idea of repeated ss more than adding a manual arp entry, even if only temporary. – Centimane Feb 14 '17 at 19:26
  • 1
    Yes, if you keep a careful eye on the output you should see it pass by. You might be better off doing while ! ss -p state syn-sent|grep 1.1.1.100;do sleep .1; done – meuh Feb 14 '17 at 19:41
  • You should add that to your answer – Centimane Feb 14 '17 at 20:27
1

You can use nethogs program that displays per-process network traffic. But arp won't be generated directly by running process it would be generated by OS. Some program may want to communicate with 1.1.1.100 and this IP doesn't exists in ARP table so ARP packet is being sent by OS to populate mac address table.

Is DHCP server running on 1.1.1.1? I would say that DHCP server probes addresses in the lease range to see which of them are free.

  • DHCP server is not running on 1.1.1.1. DHCP should only probe an address in the lease range before assigning it (so if something requested an IP, and the DHCP server was going to assign it 1.1.1.100 it would probe 1.1.1.100 first to see if it was safe to assign). – Centimane Feb 10 '17 at 13:19
  • So only one options would be dtrace, strace to find which process is trying to communicate. – Datagram.Network Feb 10 '17 at 13:29
  • I was hoping to avoid strace because there are many process trees, but it is the next thing to fall back on. I was hoping to match from ARP -> Process rather than from Process -> ARP. But it may be the only choice. – Centimane Feb 10 '17 at 14:00
  • another option is to setup dummy host/alias interface on Ethernet card. which is being arped for and check what traffic is being sent to this host, you can use nethogs to identify proccess which is sending traffic. So arp will be replied mac address added to arp table and than packet will be delivered to destination host. – Datagram.Network Feb 10 '17 at 14:10
  • Oh I like that. I'll be able to give it a shot in a bit. If you add that to your answer and my testing pans out I'll accept. – Centimane Feb 10 '17 at 14:36
1

I had this same problem just now. Having tried a few things, I came back to sysdig. This worked nicely for me:

sysdig fd.rip=1.1.1.100

In my case, the IP in question was actually 172.28.210.22, and the output was:

# sysdig fd.rip=172.28.210.22
5987580 15:42:55.952661802 7 dhclient (1232) < sendto res=300 data=...............*............RT..................................................
6318682 15:43:01.237021372 7 dhclient (1232) > sendto fd=6(<4u>172.28.210.22:67->172.28.208.42:68) size=300 tuple=0.0.0.0:68->172.28.210.22:67
6318683 15:43:01.237080305 7 dhclient (1232) < sendto res=300 data=...............*............RT..................................................
6926596 15:43:10.092470330 7 dhclient (1232) > sendto fd=6(<4u>172.28.210.22:67->172.28.208.42:68) size=300 tuple=0.0.0.0:68->172.28.210.22:67
6926608 15:43:10.092541255 7 dhclient (1232) < sendto res=300 data=...............*............RT..................................................
8882391 15:43:31.595024934 7 dhclient (1232) > sendto fd=6(<4u>172.28.210.22:67->172.28.208.42:68) size=300 tuple=0.0.0.0:68->172.28.210.22:67

which clearly showed that this was coming from dhclient.

  • This sysdig is a solid find. Their documentation could be better (The list of available fields can be obtained with 'sysdig -l'. rather than in the man page..) but it seems like a powerful tool, and distributed with Debian and Ubuntu already no less! – Centimane Dec 13 '17 at 16:14
  • Yeah, sysdig is awesome. Be sure to take a look at csysdig too.... – boyvinall Dec 13 '17 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.