1

I have some problems in a ceph cluster. The fs_apply_latency is too high which leads to high load and slow responding qemu VMs (which use ceph images as VHD). The setup is:

  • 5 hosts with 4 HDDs and 1 SSD as journal-device
  • interconnected by 3x 1 GBit bonding interface
  • separated private network for all ceph traffic

Here is the ouput of the ceph osd perf:

osd fs_commit_latency(ms) fs_apply_latency(ms) 
  0                    17                  551 
  1                     0                   18 
  2                    44                   61 
  3                     0                  915 
  4                     0                  912 
  5                    46                  646 
  6                     0                 1243 
  7                    84                  110 
  8                     1                   23 
  9                     2                  167 
 10                     0                  576 
 11                     1                 1188 
 12                     0                  105 
 13                     0                  199 
 14                     0                  536 
 15                     0                 1560 
 16                     0                  195 
 17                     0                  633 
 18                     0                  492 
 19                     0                  387

Sometimes is goes up to 3 seconds per OSD! I cannot see any heavy read or write operations on that cluster. Anyway there are blocked i/o messages from time to time for different OSDs.

Here is the momentary ceph.conf:

[global]
fsid = <removed>
mon host = mon1,mon2,mon3
mon addr = <removed>
mon initial_members = mon1,mon2,mon3
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd pool default size = 3
public network = <private>/24
cluster network = <private>/24
rbd default format = 2

[osd]
osd journal size = 10240 # useless since we use partitions nowadays
osd recovery max active = 1
osd max backfills = 1

Do you have any idea what I can do to reduce the latency? I have browsed the web for weeks and couldn't find anything really helpful.

2
  • Well, it turned out to be related to tracker.ceph.com/issues/10823. Since I have removed almost all snapshots the fs_apply_latency went dramatically down!
    – devnull
    Jan 22, 2016 at 9:00
  • I'd also add that using any distributed filesystem over 1 Gbps links should be avoided if you want good performance with lots of data. 10 Gbps links are much better. Sep 5, 2021 at 11:58

0

You must log in to answer this question.

Browse other questions tagged .