3

Answer

I fixed my "freezes" by turning autoneg of (ethtool -s eth0 duplex full autoneg off).

How I found this

Every time after a freeze, monitoring disk I/O with the help of the post below, I saw that rsynclogd was writing to disk after freezes. I then checked /var/log for the last changes and saw that kern.log had link up / link down messages after freezes. So I figured it might be the network card which is turning off & on, which looks like freezes if you're SSH'd in or trying to access your web server.

Original problem

I've got a bananapi with Xubuntu on it:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"
NAME="Ubuntu"
VERSION="14.04.3 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.3 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
No LSB modules are available.
Linux version 3.4.103 (bananapi@lemaker) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #4 SMP PREEMPT Thu Dec 18 12:55:58 CST 2014
armv7l

Every few minutes the entire machine just hangs for a few seconds. I tried running top but it doesn’t show anything abnormal cpu% wise.

I notice it's hanging because I'm ssh'd in and when I try and type sometimes it'll take anywhere from 5-30 seconds until the text appears and I can do anything on the machine.

How can I find out which process is doing this? I said it happens every few minutes, but as far as I can tell it's quite random. Sometimes I won't notice it for a while, but then again, I'm not typing on it continuously.

I tried running top as I said; it seems to be updating the list every few seconds when I'm staring at it, then it'll pause for longer and it'll "catch up" and refresh 20x in 1 second then continue as normal.

I'm also running the Apache webserver on it; sometimes the pages take ages to load -- then I know it's hanging again. It'll take anywhere from 5-30 seconds or so and it'll work fine again for a while.

[edit]

I ran posted commands in the answers below. I kept only seeing apache2 with 4096B write, 3.5% CPU but no hangs. Then it hung suddenly for about 10 seconds and the only thing that was off was rsyslogd: 0 0 |iotop 4507 40k 511B5.0%|jbd2/sda1-8 55 0 32k 0% 0 0 |iotop 4507 36k 570B 11%|| 0 0 |iotop 4507 40k 719B 11%||11%| 0 0 |iotop 4507 39k 582B 11%||rsyslogd 285 0 8192B0.5% 0 0 |iotop 4507 31k 0 12%|11%| 0 0 |iotop 4507 8653B 600B1.0%|11%| 0 0 |iotop 4507 36k 479B 11%||11%| 0 0 |iotop 4507 42k 534B 11%||11%| 0 0 |iotop 4507 37k 404B 11%||jbd2/sda1-8 55 0 8192B 0% 0 0 |iotop 4507 40k 745B 11%||11%| 0 0 |sshd: bananapi@pts/4 4549 206B 280B 0%|11%| 0 0 |iotop 4507 37k 639B7.5%|11%| 0 0 |iotop 4507 40k 533B 11%||11%| 0 0 |iotop 4507 37k 385B 11%||11%| 0 0 |iotop 4507 41k 511B 11%||11%| 0 0 |iotop 4507 7520B 0 10%|11%| 0 0 |iotop 4507 29k 555B2.5%|11%| 0 0 |iotop 4507 40k 704B 11%||rsyslogd 285 0 8192B 0% ... however not much CPU usage or disk IO from this process.

update: again the only thing that's different after it freezes is it shows rsynclogd in the I/O log.

Could this really be the source? it only writes 8192 Bytes, 0% CPU usage

[edit3] did a grep rsyslogd * in /var/log and got the following: syslog:Oct 18 04:28:10 lemaker rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="285" x-info="http://www.rsyslog.com"] rsyslogd was HUPed syslog.1:Oct 18 04:00:01 lemaker rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="267" x-info="http://www.rsyslog.com"] exiting on signal 15. syslog.1:Oct 18 04:00:43 lemaker rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="285" x-info="http://www.rsyslog.com"] start syslog.1:Oct 18 04:00:43 lemaker rsyslogd: rsyslogd's groupid changed to 104 syslog.1:Oct 18 04:00:43 lemaker rsyslogd: rsyslogd's userid changed to 101 syslog.1:Oct 18 04:00:43 lemaker rsyslogd-2039: Could no open output pipe '/dev/xconsole': No such file or directory [try http://www.rsyslog.com/e/2039 ] not sure what to make of it except it's not performing as it should

[edit5] Or could it be that it hangs and rsyslogd is just logging something about the fact that something went wrong somewhere. As in.. correlation does not equal causation? The last updated logs in /var/log: syslog, auth.log, kern.log all by user "syslog" which is I think from rsyslogd tail kern.log: Oct 18 20:43:59 lemaker kernel: [60223.243773] PHY: sunxi_gmac-0:00 - Link is Down Oct 18 20:44:12 lemaker kernel: [60236.244107] PHY: sunxi_gmac-0:00 - Link is Up - 100/Full Oct 18 20:44:22 lemaker kernel: [60246.244654] PHY: sunxi_gmac-0:00 - Link is Down Oct 18 20:44:36 lemaker kernel: [60260.245033] PHY: sunxi_gmac-0:00 - Link is Up - 100/Full Oct 18 20:45:12 lemaker kernel: [60296.246573] PHY: sunxi_gmac-0:00 - Link is Down Oct 18 20:45:25 lemaker kernel: [60309.246906] PHY: sunxi_gmac-0:00 - Link is Up - 100/Full Oct 18 20:45:57 lemaker kernel: [60341.248303] PHY: sunxi_gmac-0:00 - Link is Down Oct 18 20:46:10 lemaker kernel: [60354.248630] PHY: sunxi_gmac-0:00 - Link is Up - 100/Full Oct 18 20:46:16 lemaker kernel: [60360.249037] PHY: sunxi_gmac-0:00 - Link is Down Oct 18 20:46:30 lemaker kernel: [60374.249405] PHY: sunxi_gmac-0:00 - Link is Up - 100/Full

I reainstalled rsynclog, even disabled it. It still freezes like before. I think rsysnclog is just logging something when it freezes, but it's not the problem. As in correlation does not equal causation .

I think the problem might just be the link is up/link is down, since i notice the "Freezes" while logged in trough SSH or connecting my web server. This might make it seem like it's frozen, while it's just the network link.

  • Are any processes doing a lot of I/O? Disk reads or writes could cause this. How about memory and swap usage? – terdon Oct 18 '15 at 17:09
1

I recommend the dstat tool. It has a lot of plugins (listed via "dstat --list") that could help you narrow down where the issue is occurring.

One place to start could be with most expensive IO:

# dstat -s --top-io-adv --top-bio-adv
----swap--- -------most-expensive-i/o-process------- ----most-expensive-block-i/o-process----
 used  free|process               pid  read write cpu|process               pid  read write cpu
   0  4000M|chrome                3388  152k  97k0.6%|chrome                3388  155k  95k0.6%
   0  4000M|gnome-terminal-server 4635  586B  24k0.1%|chrome                3388    0   12k0.1%

From: http://articles.slicehost.com/2010/11/12/using-dstat-to-check-i-o-and-swap

  • I added an update to my story when I ran dstat as you suggested – Maurits Oct 18 '15 at 18:24
  • Thanks for your help, while your post wasn't the solution it surely help me identify the problem. Altough I should have checked /var/log first ;) – Maurits Oct 19 '15 at 7:27
0

The iotop utility might give you a hint, if the lag arises from hard drive or network operations.

  • iotop didnt give me anything special.... – Maurits Oct 18 '15 at 18:24
0

You should run top sorting by CPU time and taking note of all processed which have consumed more than a couple of seconds. Then, after a freeze happens, rerun top to check if a process had bubbled up to the top.

If you suspect any particular service (apache?), try to stop it and see if the issue is still present. In general, I would kill any non-critical process reported by top as an important CPU time consumer.

It would also help if you describe your hardware configuration. Did you connect any external hardware? What is the output of lsmod? Again, removing all non-essential hardware and disabling the related modules might help.

As a final note, make sure you use Ethernet when evaluating performance (or, better yet, connect to a local terminal directly). Wlan is a much more complex system with many places prone to freezes. For example, if your device loses connection and reconnects automatically, it may look like a freeze to SSH over WIFI.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.