6

Good morning, fellow *nix enthusiasts!

I have been using Debian 7 for a while now and after a recent upgrade I noticed I constantly kept running out space on my root partition. I mean to the point where I had '0' bytes left on disk! So, after a lot of searching, I was able to zero-in on the /var/log folder. I used ls -s -S to arrange the files by size in this folder and noticed that three files were GBs in size (such as 13-15 GB):

  • syslog
  • messages
  • kern.log

And yes, logrotate is working fine. It is rotating the logs. For example, I see kern.log.1 etc in /var/log. The problem is the logs are filling up so extremely fast that there's nothing logrotate can do.

Apparently, some logging process in the OS is writing a lot of data which could be because of constant errors or something(??). I don't know. All I know is my laptop is over-heating simply because there's so much processing going on all the time due to this constant write process. So, I'm losing CPU power, AND disk space.

My question is: how can I determine what process/daemon is creating this issue? How do I get to the root-cause of the problem so I could correct it? Reading these HUGE log files is not an option. Please. If I try to pull up a 15 GB log file in a text editor like leafpad or notepad on an already busy laptop, it just takes ages and ages to open. That is not practical.

I realize that this question is broad because there could be any process/daemon causing this, but I want to know if anyone has experienced this before, and if there are any usual suspects I could look at.

UPDATE:

Following Eric's advice, I arranged the files in /var/log by modification time, and 'syslog' was the last one. So, I tail'ed it. The result:

Apr 10 00:53:37 MyMachine kernel: [11608.690733]  [<ffffffffa08e4005>] ? ath9k_reg_rmw+0x35/0x70 [ath9k_htc]
Apr 10 00:53:37 MyMachine kernel: [11608.690742]  [<ffffffff81084f57>] ? process_one_work+0x147/0x3b0
Apr 10 00:53:37 MyMachine kernel: [11608.690750]  [<ffffffff81085764>] ? worker_thread+0x114/0x480
Apr 10 00:53:37 MyMachine kernel: [11608.690756]  [<ffffffff81556065>] ? __schedule+0x2e5/0x790
Apr 10 00:53:37 MyMachine kernel: [11608.690765]  [<ffffffff81085650>] ? create_worker+0x1c0/0x1c0
Apr 10 00:53:37 MyMachine kernel: [11608.690772]  [<ffffffff8108ae91>] ? kthread+0xc1/0xe0
Apr 10 00:53:37 MyMachine kernel: [11608.690780]  [<ffffffff8108add0>] ? kthread_create_on_node+0x1c0/0x1c0
Apr 10 00:53:37 MyMachine kernel: [11608.690788]  [<ffffffff8155a23c>] ? ret_from_fork+0x7c/0xb0
Apr 10 00:53:37 MyMachine kernel: [11608.690795]  [<ffffffff8108add0>] ? kthread_create_on_node+0x1c0/0x1c0
Apr 10 00:53:37 MyMachine kernel: [11608.690800] ---[ end trace 12dc8d8439345c1d ]

Unfortunately, it doesn't give me much of a hint.

  • 1
    "My question is: how can I determine what process/daemon is creating this issue?" You might start by quoting several of the most common log messages in your question. – John1024 Apr 9 '15 at 19:18
  • 1
    I'd start by tailling the files and see what has most recently been writing to them, that might be enough of a clue. – Eric Renouf Apr 9 '15 at 19:22
  • @EricRenouf Ok, I tail'ed the 'syslog' file. Please see the update. – learnerX Apr 9 '15 at 19:28
  • @learnerX that's part of a stack trace. tail -f /var/log/syslog will scroll the syslog (until it rotates) and let you see a little more of what's happening (don't post the whole thing here, but watch it and get a feel for what's repeating, so you know what to investigate). ath9k might be a wifi chipset, maybe try disabling your wifi as a troubleshooting step (also try unplugging any peripherals). – drewbenn Apr 9 '15 at 19:31
  • @learnerX Your are right that most of those messages are not helpful, but ath9k is a wifi driver. You might try disabling it. – John1024 Apr 9 '15 at 19:37
4

There is actually a strong hint in the syslog snippet you posted. The end of the line

Apr 10 00:53:37 MyMachine kernel: [11608.690733]  [<ffffffffa08e4005>] ? ath9k_reg_rmw+0x35/0x70 [ath9k_htc]

shows the stack trace is due to an unexpected error in a device driver named ath9k_htc. Hopefully, the kernel didn't panicked but the continuous repetition of errors is filling your file system.

I would then blacklist the ath9k_htc wifi driver using this command then rebooting:

echo "blacklist ath9k_htc" | sudo tee -a /etc/modprobe.d/blacklist.conf

Beware though that doing so might prevent your wifi to work if the ath9k_htc driver was nevertheless used and functional despite the errors.

You can check if a wifi device expected by the ath9k_htc driver is present in your machine by running lsusb and see if a device match one of the list available here: https://wiki.debian.org/ath9k_htc

  • Alright. I will try this and let you know. It will probably take a while since I would need to observe the disk space to see if it's filling up again. – learnerX Apr 9 '15 at 21:19
  • 5
    @learnerx no you don't. Just tail the log again and see if the repeated messages have stopped. – roaima Apr 9 '15 at 21:34
  • @roaima The problem is that this issue doesn't happen every time I turn on the laptop. It happens randomly once in 5-6 times. So, once in every 5-6 times I boot up, I see the logs filling up too quick. So, I'll have to wait and see to make sure it worked. – learnerX Apr 10 '15 at 11:00
  • @jiliagre Ok, I executed that command to blacklist ath9k_htc. After bootup, my external WiFi antenna (wlan1) doesn't work anymore. And I need it since the default laptop's wireless card (wlan0) doesn't pick up far off signals well. Waiting to see if the /var/logs have stopped filling up. – learnerX Apr 10 '15 at 11:03
  • I already updated my answer to warn you about that eventuality. The root cause is either a driver bug or a failing hardware. In the former case, make sure you have the latest update, in the latter case, things are getting more complicated. A workaround might be to monitor the syslog file size and reboot if it grows out of control, assuming the issue is not happening everytime. – jlliagre Apr 10 '15 at 11:13
3

You don't need to open the log files in an editor to see what's flooding them. Just look at the last few lines:

tail -n 999 /var/log/syslog | less

Log files from a process always contain the process ID:

Apr 10 00:00:01 harfang /USR/SBIN/CRON[345]: (root) CMD ( /usr/local/bin/midnight-stuff )
Apr 10 00:00:01 darkstar wibbled[1234]: I'm bored
Apr 10 00:00:01 darkstar wibbled[1234]: I'm still bored
Apr 10 00:00:01 darkstar wibbled[1234]: I'm bored
Apr 10 00:00:02 darkstar wibbled[1234]: I'm still bored
Apr 10 00:00:02 darkstar wibbled[1234]: I'm bored

This tells you that process 1234, which is an instance of the wibbled daemon, is producing a lot of log messages. You may want to kill it and check its configuration.

If kern.log is growing a lot, your logs aren't coming from a process but from the kernel. Flooding in the kernel logs is rarer and can be harder to pin down. It can be due to a process that's being respawned in a tight loop and is crashing immediately (perhaps due to low memory on the system). It can also be due to a buggy driver. You need to look at the messages to understand the cause.

In your case, you're seeing a backtrace from a driver. The driver is encountering a non-fatal error incessantly. Try unloading it:

rmmod ath9k

(Why ath9k? Because that's the driver that provides the function ath9k_reg_rmw, but actually because the module name would be mentioned a few lines further up from the bit you included in your question.) If the driver isn't in a module or cannot be unloaded, look for another way to disable it or stop triggering its bug; how to do that depends on what driver it is and what's wrong with it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.