2

I have noticing that, lately, my Odroid's ethernet LED is constantly blinking, even when I'm expecting no traffic. I ran iptraf and, besides the attempts of SSH brute force guess attempts (that are then blocked by fail2ban) I didn't see any TCP traffic. However, I noticed a massive amount of UDP traffic on port 123 that I have no idea where is coming from.

| UDP (46 bytes) from 188.130.254.14:443 to 192.168.1.68:123 on eth0

| UDP (46 bytes) from 188.130.254.14:443 to 192.168.1.68:123 on eth0

| UDP (46 bytes) from 121.40.223.68:20630 to 192.168.1.68:123 on eth0

| UDP (46 bytes) from 45.63.62.141:33296 to 192.168.1.68:123 on eth0

I configured my router to forward incoming UDP traffic on port 123 to an unknown IP address and still see a stupid amount of UDP packets on iptraf. Does anyone have any idea of what might be going on?

Thanks!

2

You state that

I have stopped ntpd

Then what you have been observing is nearly surely illegal activity. UDP/123 being a IANA-assigned port, it cannot be used by any other legitimate application: the Official IANA port assignment page states:

Assigned ports both System and User ports SHOULD NOT be used without or prior to IANA registration.

(System ports are defined higher up in the same document as ports in the range 0-1023).

Port TCP/123 is used by a well-known piece of malware, showing that in compromised systems where root credentials have been obtained System ports are routinely used to smuggle illicit traffic. There are many plausible reasons for using UDP instead of TCP, possibly the most important one of which is the use of an encrypted VPN (in which case wireshark will not help you in the least), and for using a System Port (it is easier to evade detection if you use an innocent port).

More than wireshark, your friend is ss:

ss -lnup | grep 123

will give you the ID of the process listening on port UDP/123. Anything but ntp, or, worse still, nothing, will mean you have been broken into. But we'll cross that bridge when we get there.

EDIT:

A follow-up to your comment. These pieces of evidence suggest you have been hacked:

  1. a mysterious service running on UDP/123, leaving no trace (which suggests the presence of a rootkit);

  2. a mysterious port-forwarding appearing on your router;

  3. connections from consumer accounts (check them on whatismyipaddress.com or with whois command). BTW, none of the three IP addresses you provided is even remotely connected with an ntp server.

Your safest bet is to re-install your operating system, then change the configuration (including password!!) of your router (possibly disabling password login altogether in favor of the use of cryptographic keys) to allow only https connections. If you do not wish to re-install the OS because you have sensitive data, then take any Linux distribution running from a USB stick (Ubuntu is just fine), boot you pc from it (not from your hard disk), install clamav, rkhunter and chkroot on the USB key, and set them to work on your hard disk. This avoids the ability of some malware to evade detection by anti-malware programs because the disk on which the malware resides is being used passively, i.e. the programs on it are not being run.

Also, remember that password protection (fail2ban notwithstanding) is not sufficient protection nowadays, and that you should always use cryptographic keys instead. Also, changing the default port for ssh connections makes you invisible at least to script kiddies (although any determined opponent will never be fooled by such a stratagem). Also, you may wish to read this post, including the answers, to get some more tips.

Good luck.

  • Thanks for the reply! The command did not return results which, according to your diagnostic, may mean I was broken into. What do you suggest as a next step? I can no longer see the traffic on iptraf. I guess the port forward I made on my router took a while to kick in. Basically what I did was forward all UDP traffic on port 123 to 192.168.2.3. On my LAN, all IPs are 192.168.1.*. I don't know why the traffic was being delivered to my Linux box in the first place. The router is supposed to block everything except the port forwards I manually define. – Mauricio Ramalho Nov 16 '17 at 11:38
  • @MauricioRamalho Pls read the Edit in my answer. – MariusMatutiae Nov 16 '17 at 19:25
  • A lot of misinformation in this answer. – symcbean Jul 10 '18 at 11:51
  • @symcbean Please feel free to write your own answer, with what you believe is more solid information. – MariusMatutiae Jul 10 '18 at 14:03
  • @MariusMatutiae: I'd just be duplicating what sourcejedi says in his/her answer. – symcbean Jul 10 '18 at 16:01
2

UDP port 123 is the default NTP port. So check your NTP configuration. The most probably it's standard synchronization. You'd have to capture packets with tcpdump and inspect the content (wireshark).

  • 1
    Hi, thanks for your reply! I have stopped ntpd. I read that UDP port 123 is the default NTP port but why are IPs from China sending UDP packets to my linux box? I don't even understand how the UDP packets are getting past my router. – Mauricio Ramalho Nov 15 '17 at 11:35
  • 2
    It's possible that your NTP client is requesting time synchronization, and those NTP servers are answering. Or, it's junk traffic, but check first to see if you are sending NTP requests out. – Jeff Schaller Nov 15 '17 at 12:02
1

I noticed a massive amount of UDP traffic on port 123 that I have no idea where is coming from.

In my definition of massive, this would mean you see traffic on port 123 consistently every second, e.g. over a whole minute.

And you say you have not requested this. For example you have not listed yourself as a public NTP server :). Or configured 5+ other computers to use this NTP server, and been watching the traffic when all 5 are turned on at the same time.

If so, there is another possibility that could explain this. Someone could be trying to use your computer as part of a flooding attack on someone else. Look up "NTP amplification attack". Top Google results include explanations from companies like Cloudflare, who sell services protecting websites from flooding attacks.

https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/

(I've seen this happen myself but with uPNP traffic, UDP port 1900, on a consumer router with a bad configuration.)

At this point, my assumption is that attackers are still trying to use you and will continue for some time. If you block them half-way through an attack, they won't necessarily get any signal that this has happened. They might notice later on, when they re-scan for amplifying NTP servers they can use.


It sounds as if your router config change just didn't work for some reason...

I guess the port forward I made on my router took a while to kick in. Basically what I did was forward all UDP traffic on port 123 to 192.168.2.3. On my LAN, all IPs are 192.168.1.*. I don't know why the traffic was being delivered to my Linux box in the first place. The router is supposed to block everything except the port forwards I manually define.

Oh. If you add this condition, I don't understand the situation, sorry. I can't understand why you would ever see these conditions, unless I start nitpicking what you've said.

Note that sometimes people configure a separate setting labelled as "DMZ" or maybe "default port forward" to a server like your Odroid, which basically forwards all ports which you don't have a specific rule for. I initially assumed you had some setup like that. If I were you, I would look for such a configuration option. It is quite plausible that I would have set this option earlier and forgotten about it.

I'm not 100% sure so far. I might also double-check that I was only seeing incoming UDP, and that I wasn't not seeing equal or greater amounts of outgoing UDP. If there is also a "massive amount" of unexpected outgoing UDP on port 123, it would suggest the Odroid had been taken over. Then you should re-install it from scratch with careful attention to security (up-to-date software, and careful about SSH access).

I think those are the most likely interpretations. The second case is less familiar to me. I haven't seen popular articles about generic server Linux distributions being compromised for flooding attacks (i.e. compromise through SSH or a web server) - and particularly not by ARM-compatible malware. All of that is possible, but I don't know how common it is.

0

may be systemd-timesyncd...

To know, use (in root) :

# ss -lnup | grep systemd-timesyn

If the return is :

users:(("systemd-timesyn",pid=xxxx,fd=xx))

So that's it.

You can also used :

systemctl status systemd-timesyncd

To stop an disable systemd-timesyncd (in root) :

# systemctl stop systemd-timesyncd

# systemctl disable systemd-timesyncd
  • Most likely not IMO, based on the port numbers. systemd-timesyncd should be contacting NTP servers which run on port 123, and getting responses back from them on port 123. But we see we are receiving packets from port 443 and from random-looking high port numbers. – sourcejedi Jul 10 '18 at 12:50
  • Note for your ss command, I think it is important to run it as root (sudo), otherwise grep will not print any results even if systemd-timesyncd is running. – sourcejedi Jul 10 '18 at 12:51
-2

You can use tcpdump to capture the traffic, then copy the pcap-file to your pc and inspect it in wireshark. Don't be fooled by the name, tcpdump captures udp packets as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.