I would like to figure out which processes are communicating with which websites over a period of time. All what I found programs like ss that list the connections that open this instant and then exit.
What I, actually, want is something like wireshark, but one that would log process names.
Is there really no such a program?
If you have a recent kernel (preferably at least 4.9, but apparently some things work at 4.2), then you can take advantage of the new dtrace facility that allows you to intercept every tcp connect() call in the kernel and show the process id, remote ip address and port.
Since this does not poll, you will not miss any short-lived connections. From the Brendan Gregg blog of 2016 typical output is
# tcpconnect
PID COMM IP SADDR DADDR DPORT
1479 telnet 4 127.0.0.1 127.0.0.1 23
1469 curl 4 10.201.219.236 54.245.105.25 80
1469 curl 4 10.201.219.236 54.67.101.145 80
1991 telnet 6 ::1 ::1 23
2015 ssh 6 fe80::2000:bff:fe82:3ac fe80::2000:bff:fe82:3ac 22
Further examples are in the bcc-tools package source. Built packages to install are available for several distributions or you can follow the compilation instructions.
You can use netstat to show tcp/ip traffic along with PID and process name, like this:
netstat -pt
To watch it live use watch -nwith an interval, like this:
watch -n 2 netstat -pt
To gather such information over time you can save and run this as a bash script:
sec=1
count=0
end=3600
echo > output.txt
while [ $count -lt $end ]; do
netstat -pt | sed -e '1,2d' >> output.txt
sleep $sec
count=$((count+1))
done
This will run netstat -pt every second for an hour and direct results into output.txt. Then you can use:
awk '{ print $7 "\t" $5 }' output.txt
to nicely print a formatted PID/process_name and a website:protocol separated by tab for each line that was gathered.
watch to monitor netstat 'live' in intervals, or craft a script that will dump netstat output periodically over a set time into a file(s) and then work the fields and info you need from it into some sort of database or table for statistics
sec=10;count=0;end=600;echo > output.txt;while [ $count -lt $end ]; do; echo $count; netstat -p >> output.tx ; echo >> output.txt; sleep $sec; count=$((count+1));done; This will run netstat -p every 10 seconds for an hour and append the output to output.txt with a new line between each repetition.
watch netstat (and you'd need to filter the output, cause it doesn't fit on the terminal screen and you end up missing some of the information)
Commented
Sep 15, 2018 at 23:39
sed and cut to filter the needed columns, like PID and Foreign Address. Then either trim or count duplicates to generate a report of which process connected with which site how many times over the hour.
At the network layer, where tcpdump sees the packets, the connection to the user process is not available. The ss tool has the option -p to list the processes associated with a network connection. As ss only shows the information at a point in time, you will have to run it often, and may still lose information about short connections.
To know what was transmitted, you can run tcpdump and capture everything, later you can filter the captured packets for the local port numbers you got from ss.
This won't give you direct information about the DNS names used, because connections are always to IP addresses. But you can search your network capture for DNS requests that respond with the IP address of the connection later used, this will let you know the DNS names used unless there are requests to different DNS names that use the same IP address.
If you really want to know what the process is doing, use strace. You will see the DNS namessent to the resolver as well as everything that is transmitted, even for short lived connections.
