find who sends spam

Question

I have a LAMP Ubuntu 10.04 server. There are many (>140) users and a lot of different web PHP sites (custom, different PHP frameworks, CMS etc).

The problem is: sometimes the server sends "spam". And local Exim isn't used for that. I discovered strange activity like the following:

/usr/bin/lsof -ni | grep smtp |grep -v ^exim4

perl      15177    www-data  510u  IPv4 1101127040      0t0  TCP server_ip:46401->65.55.37.72:smtp (SYN_SENT)
perl      15178    www-data  510u  IPv4 1101127059      0t0  TCP server_ip:51002->98.136.217.202:smtp (SYN_SENT)
perl      15179    www-data  510u  IPv4 1101126982      0t0  TCP server_ip:39232->74.125.205.26:smtp (SYN_SENT)
perl      15180    www-data  510u  IPv4 1101126975      0t0  TCP server_ip:53339->65.55.37.72:smtp (SYN_SENT)
perl      15181    www-data  510u  IPv4 1101127014      0t0  TCP server_ip:45429->65.55.37.72:smtp (SYN_SENT)
perl      15182    www-data  510u  IPv4 1101126984      0t0  TCP server_ip:49985->74.125.205.26:smtp (SYN_SENT)
perl      15183    www-data  510u  IPv4 1101126971      0t0  TCP server_ip:42199->65.55.37.72:smtp (SYN_SENT)
..........
...........
perl      15184    www-data  510u  IPv4 1101126968      0t0  TCP server_ip:36641->74.125.205.26:smtp (SYN_SENT)
perl      15186    www-data  510u  IPv4 1101126979      0t0  TCP server_ip:57690->98.138.112.32:smtp (SYN_SENT)
...........

And I can't discover who runs these Perl processes or how they are run. I tried to analyze these processes (for example pid 15179): /proc/15179/cmdline - is empty

/proc/15179/status

Name:   perl
State:  S (sleeping)
Tgid:   15179
Pid:    15179
PPid:   15176
TracerPid:  0
Uid:    33  33  33  33
Gid:    33  33  33  33
FDSize: 1024
Groups: 33 
VmPeak:    10400 kB
VmSize:    10372 kB
VmLck:         0 kB
VmHWM:      8140 kB
VmRSS:      8092 kB
VmData:     6980 kB
VmStk:        88 kB
VmExe:      1200 kB
VmLib:      1980 kB
VmPTE:        32 kB
Threads:    1
SigQ:   0/16382
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000080
SigCgt: 0000000180017427
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed:   f
Cpus_allowed_list:  0-3
Mems_allowed:   1
Mems_allowed_list:  0
voluntary_ctxt_switches:    6431
nonvoluntary_ctxt_switches: 34

lsof -n -p 15179 - here enter link description here

I tried to find the parent process: the parent pid of 15179 is 15176:

/proc/15176/cmdline - empty too

and

/proc/15176/status

Name:   perl
State:  S (sleeping)
Tgid:   15176
Pid:    15176
PPid:   1
TracerPid:  0
Uid:    33  33  33  33
Gid:    33  33  33  33
FDSize: 1024
Groups: 33 
VmPeak:    11116 kB
VmSize:    11116 kB
VmLck:         0 kB
VmHWM:      8712 kB
VmRSS:      8692 kB
VmData:     7772 kB
VmStk:        88 kB
VmExe:      1200 kB
VmLib:      1940 kB
VmPTE:        32 kB
Threads:    1
SigQ:   0/16382
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000010080
SigCgt: 0000000180007427
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed:   f
Cpus_allowed_list:  0-3
Mems_allowed:   1
Mems_allowed_list:  0
voluntary_ctxt_switches:    14467

It happens rarely (once every 2 days) and lasts a few minutes. So it's difficult to get more information. All this information is logged using a cron job which monitors the smtp connection. I have no idea how to identify who runs these processes or how they are run. Are there any tactics to find them?

Answer 1

The data you have displayed already contains a lot of info: the user's UID is 33, which on my system corresponds to www-data, and I think it is most likely the same applies to your system, because the sockets you displayed belong to www-data.

Also, I doubt the command line will bring you more information: the perl program's PPID is 15176, but 15176's PPID is 1 (i.e., init). This there is no shell, no session in between.

The IP addresses contacted are not especially worrisome: they belong to Microsoft and to Google, and those guys know how to defend themselves.

So, where is the evidence of foul play? I agree that the status SYN_SENT for the connection is indeed cause for concern, because it means your connection has not received a proper SYN/ACK, and you were left hanging.

So, what can you do to dig up more info? You cannot try to identify the user directly: your post already show that the user is www-data, and that the process is not directly connected to a terminal or session.

But you can first of all ascertain whether your IP is blacklisted, for instance here: if you are, that would be evidence of spamming.

Second, you should check your mailer's log, for anything unusual: sites refusing connection because you are on a blacklist, multiple connections from the same site, evidence of being used as a relay,....

Third, you can monitor your ports with

 ss -lntp

This tells you the pid's of processes using a (TCP) port at any given moment, checking once again for multiple connections. You can script the above command to repeat itself every second, and store its output (perhaps in conjunction with the output of users) and a timestamp, to find out more info on what is going on at the time of the suspicious connections. That can be cross-correlated post-mortem with users logged in, or users connected to your site.

For more info, you can simply dump all packets to the oft-recurring Microsoft site, something like

  nohup tcpdump -n -i eth0 host 65.52.0.0/14 -w outfile & 

The IP address range is the whole block belonging to Microsoft, as per output of whois 65.55.37.72; the above command is likely to generate quite some output, thus be prepared to hone your skills in filtering expressions with wireshark.

If all of this fails, be prepared to force your users to change their passwords.

Answer 2

Just an idea. Grep through your apache logs. If you have the time, when the mails were send it could be easy to find them. Look especially for perl scripts.

Answer 3

Get the user

Usually the uid field shows the uid of the user that launched the process.

In your case this will be the iser with the uid 33.

Use getent passwd 33 to see the name of the user.

Track the user

You can easily watch and log the activity of the user with a little C daemon,

using this little library to read the /proc/pid/status file and search after the user.

This could help you avoid problems with the server runtime.

(You can also let the daemon kill these processes)