16

So the university IT security team and I have been going around and around on this with no breaks... anyone have any thoughts on this:

I recently set up a small file server for my lab running Debian 8.6 on a dedicated computer (Intel Avoton C2550 processor -- happy to provide more hardware info if needed, but I think unnecessary). Debian installed without any problems, and at the time I also installed Samba, NTP, ZFS, and python. Things seemed to be working fine, so I let it sit and run in the corner of the lab for a few weeks.

About two weeks ago, I received an email from the IT team saying there that my server has been "compromised" and is vulnerable being used in an NTP amplification/DDoS attack (NTP Amplification Attacks Using CVE-2013-5211 as described in https://www.us-cert.gov/ncas/alerts/TA14-013A ). The sign they pointed to was a lot of NTPv2 traffic on port 123. Weirdly, the IP address that they identified this coming from (*.*.*.233) was different from the ip address my server was configured for and reported via ifconfig (*.*.*.77). Nevertheless, some basic troubleshooting revealed that my computer was indeed generating this traffic on port 123 (as revealed by tcpdump).

Here is where the bizarreness began. I first ran through the "fixes" recommended for CVE-2013-5211 (both updating the NTP past version 4.2.7 as well as disabling monlist functionality). Neither stemmed the traffic flow. I then tried blocking the UDP 123 port via IP tables:

$ /sbin/iptables -A INPUT -o eth0 -p udp --destination-port 123 -j DROP
$ /sbin/iptables -A OUTPUT -o eth0 -p udp --destination-port 123 -j DROP

but that too had no effect on the traffic. I finally tried purging NTP from the system, but that had no effect on the traffic either. As of this afternoon, nmap was still reporting:

Starting Nmap 5.51 ( http://nmap.org ) at 2016-12-19 16:15 EST
Nmap scan report for *.233
Host is up (0.0010s latency).
PORT    STATE SERVICE
123/udp open  ntp
| ntp-monlist:
|   Public Servers (2)
|       50.116.52.97    132.163.4.101
|   Public Clients (39)
|       54.90.159.15    185.35.62.119   185.35.62.233   185.35.63.86
|       54.197.89.98    185.35.62.142   185.35.62.250   185.35.63.108
|       128.197.24.176  185.35.62.144   185.35.62.251   185.35.63.128
|       180.97.106.37   185.35.62.152   185.35.63.15    185.35.63.145
|       185.35.62.27    185.35.62.159   185.35.63.27    185.35.63.146
|       185.35.62.52    185.35.62.176   185.35.63.30    185.35.63.167
|       185.35.62.65    185.35.62.186   185.35.63.34    185.35.63.180
|       185.35.62.97    185.35.62.194   185.35.63.38    185.35.63.183
|       185.35.62.106   185.35.62.209   185.35.63.39    185.35.63.185
|_      185.35.62.117   185.35.62.212   185.35.63.43

Which is all very weird since NTP has been purged from the system for weeks now.

After hitting a dead end on this path, I started thinking about the whole IP-address mismatch issue. My computer seemed to be sitting on both *.233 and *.77 IPs (as confirmed by successfully pinging both with ethernet cable attached and both unavailable with cable unplugged), but *.233 never shows up in ifconfig:

Link encap:Ethernet  HWaddr d0:XX:XX:51:78:XX  
inet addr:*.77  Bcast:*.255  Mask:255.255.255.0
inet6 addr: X::X:X:X:787a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:23023571 errors:0 dropped:1362 overruns:0 frame:0
TX packets:364849 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000 
RX bytes:7441732389 (6.9 GiB)  TX bytes:44699444 (42.6 MiB)
Memory:df300000-df37ffff 

There is no reference to *.233 in /etc/network/interfaces, so I don't see where this IP assignment is coming from.

So, I have two likely related questions I'm hoping someone can help me with: 1) How can I eliminate this NTP traffic from spewing from my server to get IT off my back? 2) What is up with this second IP address my server is sitting on and how can I remove it?

Thanks, folks :)

UPDATE: As requested: $iptables -L -v -n

Chain INPUT (policy ACCEPT 57 packets, 6540 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 27 packets, 2076 bytes)
 pkts bytes target     prot opt in     out     source               destination

And $ip addr ls

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d0:50:99:51:78:7a brd ff:ff:ff:ff:ff:ff
    inet *.77/24 brd *.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet *.167/24 brd *.255 scope global secondary dynamic eth0
       valid_lft 24612sec preferred_lft 24612sec
    inet6 X::X:X:X:787a/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether d0:50:99:51:78:7b brd ff:ff:ff:ff:ff:ff

UPDATE 2: I failed to mention that in addition to the IP address not matching up, the MAC ID also did not match. This really made me think twice about whether the traffic was indeed coming from my machine. However: (1) unplugging my server from the network made the traffic disappear; (2) move to a different network port and the traffic followed; and (3) tcpdump port 123 showed the aberrant traffic:

13:24:33.329514 IP cumm024-0701-dhcp-233.bu.edu.ntp > 183.61.254.77.44300: NTPv2, Reserved, length 440
13:24:33.329666 IP cumm024-0701-dhcp-233.bu.edu.ntp > 183.61.254.77.44300: NTPv2, Reserved, length 440
13:24:33.329777 IP cumm024-0701-dhcp-233.bu.edu.ntp > 183.61.254.77.44300: NTPv2, Reserved, length 296

UPDATE 3: $ss -uapn 'sport = :123'

State      Recv-Q Send-Q                  Local Address:Port                    Peer Address:Port 

(i.e., nothing)

$sudo cat /proc/net/dev

Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo:  327357    5455    0    0    0     0          0         0   327357    5455    0    0    0     0       0          0
  eth1:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth0: 13642399917 36270491    0 6522    0     0          0   2721337 45098276  368537    0    0    0     0       0          0

UPDATE 4: Those packets were typical of a few days ago. Today (but yes, still very high):

20:19:37.011762 IP cumm024-0701-dhcp-233.bu.edu.ntp > 103.56.63.147.26656: NTPv2, Reserved, length 152
20:19:37.011900 IP cumm024-0701-dhcp-233.bu.edu.ntp > 202.83.122.78.58066: NTPv2, Reserved, length 152
20:19:37.012036 IP cumm024-0701-dhcp-233.bu.edu.ntp > 103.56.63.147.17665: NTPv2, Reserved, length 152
20:19:37.014539 IP cumm024-0701-dhcp-233.bu.edu.ntp > 202.83.122.78.27945: NTPv2, Reserved, length 152
20:19:37.015482 IP cumm024-0701-dhcp-233.bu.edu.ntp > 202.83.122.78.42426: NTPv2, Reserved, length 152
20:19:37.015644 IP cumm024-0701-dhcp-233.bu.edu.ntp > 103.56.63.147.16086: NTPv2, Reserved, length 152

$ sudo ss -uapn '( sport = :42426 or dport = :42426 )'
State       Recv-Q Send-Q                                                        Local Address:Port                                                          Peer Address:Port 

Yes, I can ping the *.233 IP:

$ping 128.197.112.233
PING 128.197.112.233 (128.197.112.233) 56(84) bytes of data.
64 bytes from 128.197.112.233: icmp_seq=1 ttl=64 time=0.278 ms
64 bytes from 128.197.112.233: icmp_seq=2 ttl=64 time=0.282 ms
64 bytes from 128.197.112.233: icmp_seq=3 ttl=64 time=0.320 ms

No, the MAC don't match My hardware MAC address is: d0:50:99:51:78:7a The traffic is associate with MAC: bc:5f:f4:fe:a1:00

UPDATE 5: As requested, a port scan against *.233:

Starting Nmap 6.00 ( http://nmap.org ) at 2016-12-20 20:38 EET
NSE: Loaded 17 scripts for scanning.
Initiating SYN Stealth Scan at 20:38
Scanning cumm024-0701-dhcp-233.bu.edu (128.197.112.233) [1024 ports]
Discovered open port 22/tcp on 128.197.112.233
Completed SYN Stealth Scan at 20:38, 9.79s elapsed (1024 total ports)
Initiating Service scan at 20:38
Scanning 1 service on cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Completed Service scan at 20:38, 0.37s elapsed (1 service on 1 host)
Initiating OS detection (try #1) against cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Initiating Traceroute at 20:38
Completed Traceroute at 20:38, 0.10s elapsed
NSE: Script scanning 128.197.112.233.

[+] Nmap scan report for cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Host is up (0.083s latency).
Not shown: 1013 filtered ports

PORT    STATE  SERVICE VERSION
21/tcp  closed ftp
22/tcp  open   ssh     OpenSSH 5.5p1 Debian 6+squeeze1 (protocol 2.0)
23/tcp  closed telnet
25/tcp  closed smtp
43/tcp  closed whois
80/tcp  closed http
105/tcp closed unknown
113/tcp closed ident
210/tcp closed z39.50
443/tcp closed https
554/tcp closed rtsp

Device type: general purpose
Running: Linux 2.6.X
OS CPE: cpe:/o:linux:kernel:2.6
OS details: DD-WRT v24-sp2 (Linux 2.6.19)
Uptime guess: 45.708 days (since Sat Nov  5 03:39:36 2016)
Network Distance: 9 hops
TCP Sequence Prediction: Difficulty=204 (Good luck!)
IP ID Sequence Generation: All zeros
Service Info: OS: Linux; CPE: cpe:/o:linux:kernel


TRACEROUTE (using port 25/tcp)
HOP RTT      ADDRESS
1   0.95 ms  router1-lon.linode.com (212.111.33.229)
2   0.70 ms  109.74.207.0
3   1.09 ms  be4464.ccr21.lon01.atlas.cogentco.com (204.68.252.85)
4   1.00 ms  be2871.ccr42.lon13.atlas.cogentco.com (154.54.58.185)
5   63.45 ms be2983.ccr22.bos01.atlas.cogentco.com (154.54.1.178)
6   63.60 ms TrusteesOfBostonUniversity.demarc.cogentco.com (38.112.23.118)
7   63.55 ms comm595-core-res01-gi2-3-cumm111-bdr-gw01-gi1-2.bu.edu (128.197.254.125)
8   63.61 ms cumm024-dist-aca01-gi5-2-comm595-core-aca01-gi2-2.bu.edu (128.197.254.206)
9   90.28 ms cumm024-0701-dhcp-233.bu.edu (128.197.112.233)

OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .

Nmap done: 1 IP address (1 host up) scanned in 20.73 seconds
           Raw packets sent: 557 (25.462KB) | Rcvd: 97 (8.560KB)

and on UDP:

Starting Nmap 6.00 ( http://nmap.org ) at 2016-12-20 20:44 EET
NSE: Loaded 17 scripts for scanning.
Initiating Ping Scan at 20:44
Scanning 128.197.112.233 [4 ports]
Completed Ping Scan at 20:44, 1.10s elapsed (1 total hosts)
Initiating UDP Scan at 20:44
Scanning cumm024-0701-dhcp-233.bu.edu (128.197.112.233) [1024 ports]
Completed UDP Scan at 20:44, 6.31s elapsed (1024 total ports)
Initiating Service scan at 20:44
Scanning 1024 services on cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Service scan Timing: About 0.39% done
Service scan Timing: About 3.12% done; ETC: 22:12 (1:25:46 remaining)
Service scan Timing: About 6.05% done; ETC: 21:53 (1:04:39 remaining)
Service scan Timing: About 8.98% done; ETC: 21:46 (0:56:03 remaining)
Discovered open port 123/udp on 128.197.112.233
Discovered open|filtered port 123/udp on cumm024-0701-dhcp-233.bu.edu (128.197.112.233) is actually open
Completed Service scan at 21:31, 2833.50s elapsed (1024 services on 1 host)
Initiating OS detection (try #1) against cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Retrying OS detection (try #2) against cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
NSE: Script scanning 128.197.112.233.
Initiating NSE at 21:31
Completed NSE at 21:31, 10.02s elapsed

[+] Nmap scan report for cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Host is up (0.089s latency).
Not shown: 1023 open|filtered ports

PORT    STATE SERVICE VERSION
123/udp open  ntp?

1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at http://www.insecure.org/cgi-bin/servicefp-submit.cgi :
SF-Port123-UDP:V=6.00%I=7%D=12/20%Time=58597D5C%P=x86_64-unknown-linux-gnu
SF:%r(NTPRequest,30,"\xe4\x02\x04\xee\0\0\x8a\xff\0:t\xd9\x84\xa3\x04e\xdb
SF:\xcaeEX\xdbC'\xc5O#Kq\xb1R\xf3\xdc\x03\xfb\xb8\+>U\xab\xdc\x03\xfb\xb8\
SF:+T\xd1\xe9")%r(Citrix,C,"\xde\xc0\x010\x02\0\xa8\xe3\0\0\0\0");

Too many fingerprints match this host to give specific OS details
Network Distance: 9 hops

OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .

Nmap done: 1 IP address (1 host up) scanned in 2863.89 seconds
           Raw packets sent: 175 (6.720KB) | Rcvd: 50 (10.088KB)
  • 3
    Sounds like you're machine is compromised and trying to hide that fact. What is the full output of iptables -L -v -n and ip addr ls. – Mark Wagner Dec 19 '16 at 23:02
  • 2
    @TimOtchy Just to clarify, when you disconnect the ethernet cable to the server, are you doing it at the back of the server, or at the switch? Do you have a spare switch and you could run an ethernet cable from the server to the switch but have nothing else (apart from power) plugged into the switch. The idea is to have the link enabled on the server but it otherwise isolated, and see if *233 is reachable. – icarus Dec 20 '16 at 17:18
  • 3
    Given (server->switch no_connection LAN) and you can ping both from server, and (server no_connection LAN) and you can only ping *.77 from server, I think we can conclude that server is also serving *.233. Next question, is the "server" a "big" box? I am thinking that maybe it has a separate chassis management CPU or maybe it is an x86 device and has an inbuilt monitoring device. I would be inclined to run a full port scan against *.233 and see what ports are open. – icarus Dec 20 '16 at 18:27
  • 2
    See en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface this is a separate CPU, nothing to do with the main cpu, the bios or the main OS. – icarus Dec 20 '16 at 20:47
  • 2
    Looks like the fix ought to address the BMC (IPMI) processor, which is running ssh (and linux according the the guess from nmap). The Asrock site has software from 23rd March 2016 on it, but my guess is that that will not help. My thought is to see if you can ssh into the *233 address. I would guess that you would need to set up a username and password, perhaps in the BIOS settings under BMC options??? – icarus Dec 20 '16 at 21:01
12
+100

This is a server class machine with IPMI. The "ghost" NTP server that is causing the issue is running on the BMC processor on the system and not the main CPU.

| improve this answer | |
  • Just made it through 24 hrs without any new NTP traffic from this mac/IP. Absolutely was the BMC processor running a very old firmware version and making use of an NTP package still vulnerable to the amplification exploit. Thank you for all your help in fixing this! – Tim Otchy Dec 23 '16 at 18:29
8

As it has already been said, your IPMI BMC is (probably) the problem. If you cannot get to the web interface or ssh interface of the IPMI interface, you can try to get access using IPMI Tool on your Debian installation:

apt install ipmitool

Once installed, you can pass commands to the BMC like this (if it is on port 1):

ipmitool lan set 1 ipaddr 192.168.1.211

Here is a resource for IPMI network configuration with IPMITool. man ipmitool is always a good read if you get stuck...

You may reset the BMC if you need to.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.