So the university IT security team and I have been going around and around on this with no breaks... anyone have any thoughts on this:
I recently set up a small file server for my lab running Debian 8.6 on a dedicated computer (Intel Avoton C2550 processor -- happy to provide more hardware info if needed, but I think unnecessary). Debian installed without any problems, and at the time I also installed Samba, NTP, ZFS, and python. Things seemed to be working fine, so I let it sit and run in the corner of the lab for a few weeks.
About two weeks ago, I received an email from the IT team saying there that my server has been "compromised" and is vulnerable being used in an NTP amplification/DDoS attack (NTP Amplification Attacks Using CVE-2013-5211 as described in https://www.us-cert.gov/ncas/alerts/TA14-013A ). The sign they pointed to was a lot of NTPv2 traffic on port 123. Weirdly, the IP address that they identified this coming from (
*.*.*.233) was different from the ip address my server was configured for and reported via ifconfig (
*.*.*.77). Nevertheless, some basic troubleshooting revealed that my computer was indeed generating this traffic on port 123 (as revealed by tcpdump).
Here is where the bizarreness began. I first ran through the "fixes" recommended for CVE-2013-5211 (both updating the NTP past version 4.2.7 as well as disabling monlist functionality). Neither stemmed the traffic flow. I then tried blocking the UDP 123 port via IP tables:
$ /sbin/iptables -A INPUT -o eth0 -p udp --destination-port 123 -j DROP $ /sbin/iptables -A OUTPUT -o eth0 -p udp --destination-port 123 -j DROP
but that too had no effect on the traffic. I finally tried purging NTP from the system, but that had no effect on the traffic either. As of this afternoon, nmap was still reporting:
Starting Nmap 5.51 ( http://nmap.org ) at 2016-12-19 16:15 EST Nmap scan report for *.233 Host is up (0.0010s latency). PORT STATE SERVICE 123/udp open ntp | ntp-monlist: | Public Servers (2) | 126.96.36.199 188.8.131.52 | Public Clients (39) | 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 | 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 | 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 | 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 | 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 | 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 | 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 | 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 | 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 |_ 184.108.40.206 220.127.116.11 18.104.22.168
Which is all very weird since NTP has been purged from the system for weeks now.
After hitting a dead end on this path, I started thinking about the whole IP-address mismatch issue. My computer seemed to be sitting on both *.233 and *.77 IPs (as confirmed by successfully pinging both with ethernet cable attached and both unavailable with cable unplugged), but *.233 never shows up in ifconfig:
Link encap:Ethernet HWaddr d0:XX:XX:51:78:XX inet addr:*.77 Bcast:*.255 Mask:255.255.255.0 inet6 addr: X::X:X:X:787a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:23023571 errors:0 dropped:1362 overruns:0 frame:0 TX packets:364849 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:7441732389 (6.9 GiB) TX bytes:44699444 (42.6 MiB) Memory:df300000-df37ffff
There is no reference to *.233 in /etc/network/interfaces, so I don't see where this IP assignment is coming from.
So, I have two likely related questions I'm hoping someone can help me with: 1) How can I eliminate this NTP traffic from spewing from my server to get IT off my back? 2) What is up with this second IP address my server is sitting on and how can I remove it?
Thanks, folks :)
$iptables -L -v -n
Chain INPUT (policy ACCEPT 57 packets, 6540 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 27 packets, 2076 bytes) pkts bytes target prot opt in out source destination
$ip addr ls
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether d0:50:99:51:78:7a brd ff:ff:ff:ff:ff:ff inet *.77/24 brd *.255 scope global eth0 valid_lft forever preferred_lft forever inet *.167/24 brd *.255 scope global secondary dynamic eth0 valid_lft 24612sec preferred_lft 24612sec inet6 X::X:X:X:787a/64 scope link valid_lft forever preferred_lft forever 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether d0:50:99:51:78:7b brd ff:ff:ff:ff:ff:ff
I failed to mention that in addition to the IP address not matching up, the MAC ID also did not match. This really made me think twice about whether the traffic was indeed coming from my machine. However: (1) unplugging my server from the network made the traffic disappear; (2) move to a different network port and the traffic followed; and (3)
tcpdump port 123 showed the aberrant traffic:
13:24:33.329514 IP cumm024-0701-dhcp-233.bu.edu.ntp > 22.214.171.124.44300: NTPv2, Reserved, length 440 13:24:33.329666 IP cumm024-0701-dhcp-233.bu.edu.ntp > 126.96.36.199.44300: NTPv2, Reserved, length 440 13:24:33.329777 IP cumm024-0701-dhcp-233.bu.edu.ntp > 188.8.131.52.44300: NTPv2, Reserved, length 296
$ss -uapn 'sport = :123'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
$sudo cat /proc/net/dev
Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed lo: 327357 5455 0 0 0 0 0 0 327357 5455 0 0 0 0 0 0 eth1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth0: 13642399917 36270491 0 6522 0 0 0 2721337 45098276 368537 0 0 0 0 0 0
UPDATE 4: Those packets were typical of a few days ago. Today (but yes, still very high):
20:19:37.011762 IP cumm024-0701-dhcp-233.bu.edu.ntp > 184.108.40.206.26656: NTPv2, Reserved, length 152 20:19:37.011900 IP cumm024-0701-dhcp-233.bu.edu.ntp > 220.127.116.11.58066: NTPv2, Reserved, length 152 20:19:37.012036 IP cumm024-0701-dhcp-233.bu.edu.ntp > 18.104.22.168.17665: NTPv2, Reserved, length 152 20:19:37.014539 IP cumm024-0701-dhcp-233.bu.edu.ntp > 22.214.171.124.27945: NTPv2, Reserved, length 152 20:19:37.015482 IP cumm024-0701-dhcp-233.bu.edu.ntp > 126.96.36.199.42426: NTPv2, Reserved, length 152 20:19:37.015644 IP cumm024-0701-dhcp-233.bu.edu.ntp > 188.8.131.52.16086: NTPv2, Reserved, length 152 $ sudo ss -uapn '( sport = :42426 or dport = :42426 )' State Recv-Q Send-Q Local Address:Port Peer Address:Port
Yes, I can ping the *.233 IP:
$ping 184.108.40.206 PING 220.127.116.11 (18.104.22.168) 56(84) bytes of data. 64 bytes from 22.214.171.124: icmp_seq=1 ttl=64 time=0.278 ms 64 bytes from 126.96.36.199: icmp_seq=2 ttl=64 time=0.282 ms 64 bytes from 188.8.131.52: icmp_seq=3 ttl=64 time=0.320 ms
No, the MAC don't match My hardware MAC address is: d0:50:99:51:78:7a The traffic is associate with MAC: bc:5f:f4:fe:a1:00
UPDATE 5: As requested, a port scan against *.233:
Starting Nmap 6.00 ( http://nmap.org ) at 2016-12-20 20:38 EET NSE: Loaded 17 scripts for scanning. Initiating SYN Stealth Scan at 20:38 Scanning cumm024-0701-dhcp-233.bu.edu (184.108.40.206) [1024 ports] Discovered open port 22/tcp on 220.127.116.11 Completed SYN Stealth Scan at 20:38, 9.79s elapsed (1024 total ports) Initiating Service scan at 20:38 Scanning 1 service on cumm024-0701-dhcp-233.bu.edu (18.104.22.168) Completed Service scan at 20:38, 0.37s elapsed (1 service on 1 host) Initiating OS detection (try #1) against cumm024-0701-dhcp-233.bu.edu (22.214.171.124) Initiating Traceroute at 20:38 Completed Traceroute at 20:38, 0.10s elapsed NSE: Script scanning 126.96.36.199. [+] Nmap scan report for cumm024-0701-dhcp-233.bu.edu (188.8.131.52) Host is up (0.083s latency). Not shown: 1013 filtered ports PORT STATE SERVICE VERSION 21/tcp closed ftp 22/tcp open ssh OpenSSH 5.5p1 Debian 6+squeeze1 (protocol 2.0) 23/tcp closed telnet 25/tcp closed smtp 43/tcp closed whois 80/tcp closed http 105/tcp closed unknown 113/tcp closed ident 210/tcp closed z39.50 443/tcp closed https 554/tcp closed rtsp Device type: general purpose Running: Linux 2.6.X OS CPE: cpe:/o:linux:kernel:2.6 OS details: DD-WRT v24-sp2 (Linux 2.6.19) Uptime guess: 45.708 days (since Sat Nov 5 03:39:36 2016) Network Distance: 9 hops TCP Sequence Prediction: Difficulty=204 (Good luck!) IP ID Sequence Generation: All zeros Service Info: OS: Linux; CPE: cpe:/o:linux:kernel TRACEROUTE (using port 25/tcp) HOP RTT ADDRESS 1 0.95 ms router1-lon.linode.com (184.108.40.206) 2 0.70 ms 220.127.116.11 3 1.09 ms be4464.ccr21.lon01.atlas.cogentco.com (18.104.22.168) 4 1.00 ms be2871.ccr42.lon13.atlas.cogentco.com (22.214.171.124) 5 63.45 ms be2983.ccr22.bos01.atlas.cogentco.com (126.96.36.199) 6 63.60 ms TrusteesOfBostonUniversity.demarc.cogentco.com (188.8.131.52) 7 63.55 ms comm595-core-res01-gi2-3-cumm111-bdr-gw01-gi1-2.bu.edu (184.108.40.206) 8 63.61 ms cumm024-dist-aca01-gi5-2-comm595-core-aca01-gi2-2.bu.edu (220.127.116.11) 9 90.28 ms cumm024-0701-dhcp-233.bu.edu (18.104.22.168) OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 20.73 seconds Raw packets sent: 557 (25.462KB) | Rcvd: 97 (8.560KB)
and on UDP:
Starting Nmap 6.00 ( http://nmap.org ) at 2016-12-20 20:44 EET NSE: Loaded 17 scripts for scanning. Initiating Ping Scan at 20:44 Scanning 22.214.171.124 [4 ports] Completed Ping Scan at 20:44, 1.10s elapsed (1 total hosts) Initiating UDP Scan at 20:44 Scanning cumm024-0701-dhcp-233.bu.edu (126.96.36.199) [1024 ports] Completed UDP Scan at 20:44, 6.31s elapsed (1024 total ports) Initiating Service scan at 20:44 Scanning 1024 services on cumm024-0701-dhcp-233.bu.edu (188.8.131.52) Service scan Timing: About 0.39% done Service scan Timing: About 3.12% done; ETC: 22:12 (1:25:46 remaining) Service scan Timing: About 6.05% done; ETC: 21:53 (1:04:39 remaining) Service scan Timing: About 8.98% done; ETC: 21:46 (0:56:03 remaining) Discovered open port 123/udp on 184.108.40.206 Discovered open|filtered port 123/udp on cumm024-0701-dhcp-233.bu.edu (220.127.116.11) is actually open Completed Service scan at 21:31, 2833.50s elapsed (1024 services on 1 host) Initiating OS detection (try #1) against cumm024-0701-dhcp-233.bu.edu (18.104.22.168) Retrying OS detection (try #2) against cumm024-0701-dhcp-233.bu.edu (22.214.171.124) NSE: Script scanning 126.96.36.199. Initiating NSE at 21:31 Completed NSE at 21:31, 10.02s elapsed [+] Nmap scan report for cumm024-0701-dhcp-233.bu.edu (188.8.131.52) Host is up (0.089s latency). Not shown: 1023 open|filtered ports PORT STATE SERVICE VERSION 123/udp open ntp? 1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at http://www.insecure.org/cgi-bin/servicefp-submit.cgi : SF-Port123-UDP:V=6.00%I=7%D=12/20%Time=58597D5C%P=x86_64-unknown-linux-gnu SF:%r(NTPRequest,30,"\xe4\x02\x04\xee\0\0\x8a\xff\0:t\xd9\x84\xa3\x04e\xdb SF:\xcaeEX\xdbC'\xc5O#Kq\xb1R\xf3\xdc\x03\xfb\xb8\+>U\xab\xdc\x03\xfb\xb8\ SF:+T\xd1\xe9")%r(Citrix,C,"\xde\xc0\x010\x02\0\xa8\xe3\0\0\0\0"); Too many fingerprints match this host to give specific OS details Network Distance: 9 hops OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 2863.89 seconds Raw packets sent: 175 (6.720KB) | Rcvd: 50 (10.088KB)