0

i installed new debian 9.8 server for DB application.it has two disk and running on vCenter 6.0.

First write disk write speed 100-120 Mb/sec suddenly after for while disk write speed decrease drastically to 1-5 Mb/sec.

  • i change datastore totaly different storage- same problem continued i
  • updated kernel to new one(5.0.14) nothing happened
  • i added no barrier=0 noatime on fstab; same problem continued
  • umount disk and fsck ; same problem continued

this mount output;

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32686212k,nr_inodes=53329,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6539196k,mode=755)
/dev/mapper/DPADBSRV1--vg-root on / type ext4 (rw,noatime,nobarrier,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12165)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
/dev/sdb1 on /data type ext4 (rw,noatime,nobarrier)
/dev/sda1 on /boot type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=2344888k,mode=700,uid=1000,gid=1000)

This is the dd wirte test within one hour period.

134217728 bytes (134 MB, 128 MiB) copied, 1.34163 s, 100 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 1.43202 s, 93.7 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 1.40082 s, 95.8 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 1.31636 s, 102 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 1.39147 s, 96.5 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 1.81849 s, 73.8 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.4479 s, 65.3 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 1.40814 s, 95.3 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 9.36281 s, 14.3 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 3.23355 s, 41.5 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 35.4718 s, 3.8 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 33.546 s, 4.0 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 61.1867 s, 2.2 MB/s
root@DP1:/home/debian# dd if=/dev/zero of=tempfile bs=1M count=128
  • Assuming SSDs, a guess: After some time the fast internal cache of the SSD fills (which they use to measure the write speed they use for adverising), and you start see the "real" write speed. – dirkt May 10 at 9:26
0

Don't know if it will help you but I had serious issues on a HP server and "non certified HP SSD disks" configured in RAID. When transferring huge files, the speed was ok for 25Gb then falling down to 10-15Mb/sec without any explanation. And when I say "non certified disks", it was still Samsung 850 PRO 1Tb SSD...

I tried every possible configuration

  • raid card options
  • drivers
  • firmwares
  • kernel

The only solution I found is to plug those 2 disks on the SATA controller of the mainboard and create a software RAID.

And on the same RAID controller, everything was fine with the 2 SSD shipped with the server...

  • Just a couple of months ago we wanted to add 2TB disks to our ceph cluster, but the performance was so bad we couldn't even fully integrate them. Then we tested those disks independent of any service, it turned out those disks couldn't stand permanent read/writes. After a couple of minutes the performance dropped from 20 MB/s to 3 MB/s, sometimes even less. Research about those specific disks confirmed what we saw in our tests, so in conclusion you'll have to carefully choose your disks for specific workloads. – eblock May 10 at 7:08
0

Long time research and digging on google's deep depth finally find the problem.But i'm do not what is root cause. The problem was CPU.Firstly we create the VM 2*4 vCPU then today i read about Debian bug forum archive about Intel(R) Xeon(R) CPU E5-2xxx family then the holy moment dunk my head. Reduce the vCPU 1x1. Problem solved.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.