5

I got a strange issue with extremely slow dnf, to the point where computer freezes for a few seconds during installation of packages. Running Fedora 29, but this is happening for years. Otherwise the computer works fine, never crashes, obviously only dnf hits some weak spot. Not only can "dnf upgrade" take an hour for 100 packages, every year when I upgrade Fedora, I wait for Christmas holidays so I can live without computer over the weekend as 3000+ package update starts on Friday evening and finishes on Sunday.

Root directory is mounted to an old SSD disk, /home on a new classic disk (WD Caviar Black), there is another disk just for backups (just to tell there are multiple disks installed). I have suspected the old SSD to misbehave, moved the root partition to one of the classic disks and it did not improve the situation (was even slower).

Journalctl or dmesg or /var/log/messages do not show anything special. Sysbench says that disk performance of every installed disk is fine, just as fine as on another (older) computer that also runs Fedora 29 and where dnf is fine (upgrade in a few minutes, annual release upgrade during lunch).

Does anyone have an idea where to look for what is happening?

P.S. top, iotop, iostat, dmesg, journalctl, such elementary things were checked a lot. iotop shows 99.99% activity on disk by python... (dnf) when it does the upgrade but I guess that's normal. Even sysbench (as stated above) shows the disks are a bit faster than at another computer that does dnf in a breeze.

For the make: i7-920 processor, 24 G RAM, no swap

Improve this question
5
  • Just to be sure: this is during the installation part, and not network?
    – mattdm
    Commented Jan 26, 2019 at 9:58
  • Can you try / have you tried running dnf through strace to see if that gives any clues? Have you checked the dnf-specific log files, /var/log/dnf*.log? (These contain some stuff that isn't sent to the journal).
    – Adam Williamson
    Commented Jan 26, 2019 at 11:06
  • Is it cpu-bound or i/o bound? What's the utilization of the CPU and the disks (iostat -x)? (Long shot, but could try rpm --rebuilddb.)
    – fche
    Commented Jan 26, 2019 at 14:48
  • @mattdm It's just installation, not network. Even on release upgrade, when all packages are downloaded and the computers is installing them in single user mode, it takes a long time. So network or any other software interference seems to be irrelevant.
    – Robert Špendl
    Commented Jan 26, 2019 at 15:27
  • I have tried perf as @Joe Mario suggested and fdatasync sems to be very slow. Any ideas on "step 2"?
    – Robert Špendl
    Commented Jan 26, 2019 at 15:41

2 Answers 2

Reset to default
1

Robert: It would be interesting to know what system call is taking the longest.

Can you install a simple (random) package using perf trace and then sort to find the longest calls?

For example: # perf trace -s -o /tmp/trace.out dnf install -y xorg-x11-apps-7.7-20.fc28.x86_64

And then post the contents of your /tmp/trace.out file. It's a summary file, so it shouldn't be too large, but it will help point you at what syscall(s) are taking the longest, and if they are way out of normal range.

Improve this answer
11
  • Seems it's getting warmer. I have tried your suggestion on both "slow" and "normal" computers and it looks like fdatasync is the culprit. SLOW ` syscall calls total min avg max stddev (msec) (msec) (msec) (msec) (%) --------------- -------- --------- --------- --------- --------- ------ fdatasync 69 20651.658 0.072 299.299 6356.138 37.69% ` NORMAL fdatasync 68 317.992 0.065 4.676 82.016 29.37% `
    – Robert Špendl
    Commented Jan 26, 2019 at 15:38
  • (eventually I'll get better at editing comments, have patience with a noob)
    – Robert Špendl
    Commented Jan 26, 2019 at 15:39
  • It takes fdatasync on average 300 ms on the slow system and less than 5 ms on the normal.
    – Robert Špendl
    Commented Jan 26, 2019 at 15:40
  • I have tried a simple fsync benchmark github.com/ncw/fsyncbench Every disk on both slow & normal computers was OK, SSDs performing better (6-10 ms/fsync) and classid discs 20-40 ms/fsync. Any idea what kind of fsync should I try to find what exactly is causing such delays?
    – Robert Špendl
    Commented Jan 26, 2019 at 15:57
  • An individual fdatasync should not take anywhere that long to an SSD. Is it worn out? (smartctl -a /dev/sdXYZ)
    – fche
    Commented Jan 26, 2019 at 16:13
0

I know this question was from a long time ago, but I believe the cause of this (it was for me) may be the use of SMR (Single Magnetic Resonance) drives, which have ridiculously slow write speeds. Here are some references:

and some vendor references:

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .