As on the details below it look like that the cause of the problem is the
pmlogger is spawning the find command. And its takes a long time for the
find to scan all the files in one of my partition. And it happens that it is my backup partition, so it is expected there are so many files there.
Honestly this was not happening before, I am just getting a burst read io for a few minutes and will stop, but now it takes hours.
As requested I will change my question.
How should I exclude a directory or partition to pmlogger? Cause I don't know if it's good idea to disable the whole
pmlogger and also I don't know why
pmlogger doing this. So if I can make to exclude my backup directory to
pmlogger maybe I can narrow more the problem.
I forgot when exactly it started, If I am correct that was when I update all packages last week.
On iotop the command is
find . -daystart -name NoSuchFile, I dont know what it does to the files it reading, when I tried to use
fatrace it open a lot of files per second.
Also SELinux keeps telling about this
SELinux is preventing find from open access on the directory /sys/kernel/config. and some read access on irq, dev, usermodehelper, net, rpc, and pcp.WXW8e2p7o.
Anyone can give me some light on this and how should I deal it?
This will just happen after boot up, then will last an hour or so.
Then when I execute
sudo fatrace | grep '(7290)' *7290 is the find pid. it will log this
find(7290): RC /mnt/Time Machine/... find(7290): O /mnt/Time Machine/... find(7290): R /mnt/Time Machine/... and so on...
BTW when I execute that, at that time it accessing my laptop second hard drive(Ext4). this hard drive I think it quite old around 2 and half years, But I checked the Overall assessment on SMART Data & Self-Test is still OK.
Here is the output of free
total used free shared buff/cache available Mem: 12140160 1820112 3253992 266344 7066056 9736188 Swap: 6143996 0 6143996
And here is the output of vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 1 0 3256144 4322284 2770984 0 0 893 28 761 1914 8 4 70 18 0
I found pstree command, and found this
And I check the pmlogger on systmectl
● pmlogger.service - Performance Metrics Archive Logger Loaded: loaded (/usr/lib/systemd/system/pmlogger.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2018-12-27 11:37:16 PST; 38min ago Docs: man:pmlogger(1) Process: 1935 ExecStart=/usr/share/pcp/lib/pmlogger start (code=exited, status=0/SUCCESS) Main PID: 7150 (pmlogger) Tasks: 5 (limit: 4915) Memory: 6.6G CGroup: /system.slice/pmlogger.service ├─2043 /usr/bin/sh /usr/share/pcp/lib/pmlogger start ├─2047 /usr/bin/sh /usr/libexec/pcp/bin/pmlogger_check -V ├─7150 /usr/libexec/pcp/bin/pmlogger -P -r -T24h10m -c config.default -v 100mb -m pmlogger_check 20181227.11.37 ├─7264 /usr/bin/sh /usr/libexec/pcp/bin/pmlogger_daily -K -V └─7290 find . -daystart -name NoSuchFile Dec 27 11:37:05 fedora-acer systemd: Starting Performance Metrics Archive Logger... Dec 27 11:37:05 fedora-acer pmlogger: Starting pmlogger ... Dec 27 11:37:05 fedora-acer systemd: pmlogger.service: Can't open PID file /run/pcp/pmlogger.pid (yet?) after start: No such file or directory Dec 27 11:37:16 fedora-acer systemd: pmlogger.service: Supervising process 7150 which is not our child. We'll most likely not notice when it exits. Dec 27 11:37:16 fedora-acer systemd: Started Performance Metrics Archive Logger.