1

I'm doing some intensive testing on a mdadm (software RAID) partition made of 6 SSDs using fio (Ubuntu 16.10 Server). The partition was formatted using XFS, is unmounted during tests, contains no files and isn't a boot partition. It happens that sometimes, after a cold start or a reboot, Ubuntu can't complete the boot procedure and stalls on this message:

[  OK  ] Started File System Check on /dev/disk/by-uuid/8332-B8BE.
[FAILED] Failed to mount /mnt/raid10.
See 'systemctl status mnt-raid10.mount' for details.

This is what I get after entering in the recovery mode and issuing systemctl status mnt-raid10.mount:

mnt-raid10.mount - /mnt/raid10
 Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
 Active: failed (Result: exit-code) since mer 2017-02-15 08:18:33 CET; 1h 28min ago
  Where: /mnt/raid10
   What: /dev/md2p1
   Docs: man:fstab(5)
         man:systemd-fstab-generator(8)

systemd[1]: Mounting /mnt/raid10...
mount[2064]: mount: mount /dev/md2p1 on /mnt/raid10 failed: La struttura necessita di una pulizia
mnt-raid10.mount: Mount process exited, code=exited status=32
Failed to mount /mnt/raid10.
mnt-raid10.mount: Unit entered failed state.

If I launch xfs_repair /dev/md2p1 Ubuntu boots correctly and I can mount the partition.

The same problem may happen also if I try to mount the partition after testing and before rebooting by executing mount -t xfs /dev/md2p1 /mnt/raid10.

I wonder if this is a XFS known issue, a mdadm issue or simply its due to fio raw writing on the md device.

fio configuration

Here is a configuration file (config.ini) that I'm using for 4k 100% read/write mix:

[4k]
filename=/dev/md2p1
rw=randrw
blocksize=4k
filename=test.bin
ioengine=libaio
iodepth=16
direct=1
invalidate=1
runtime=60
write_bw_log=time_series
write_lat_log=time_series
write_iops_log=time_series
log_avg_msec=1000
numjobs=1
rwmixread=100
refill_buffers=1
norandommap=1
randrepeat=0
per_job_logs=0
description="some text"

and here's how I call fio:

fio config.ini > log.txt

log.txt, config.ini and test.bin are located on a different disk device (my home directory, which is a software RAID1 partition on HDDs mounted using /dev/md1).

  • What do you mean by "fio raw writing on the md device"? Are you using fio to benchmark the write performance of the raw md device after building a filesystem on it? – Andrew Henle Feb 15 '17 at 11:29
  • Also, XFS pretty much requires file system barriers to work. See this older LWN article: lwn.net/Articles/283161 Are your XFS file system mounted with file system barriers enabled? If so, are you seeing any log entries anywhere indicating file system barriers have been disabled for some reason? – Andrew Henle Feb 15 '17 at 11:42
  • @AndrewHenle The device is formatted as XFS but is unmounted during testing. For this reason I guess fio writes "raw" data (doesn't pass through the FS level) and so corrupts the FS. I'm using fio to read and write 4k and 8k chunk sizes. Also, I didn't specify any mount option except from defaults. – pietrop Feb 15 '17 at 12:04
  • 1
    Post your fio configuration and the command you run the test. If you are writing to the raw disk ( /dev/md2p1 ) it doesn't matter if the filesystem is mounted or not, the filesystem will be trashed after your test. – Thomas Feb 15 '17 at 17:33
  • @Thomas I've updated my question. Thank you all – pietrop Feb 16 '17 at 13:33
1

You are running your fio benchmark against the raw blockdevice /dev/md2p1, which thrashes your filesystem information.

To avoid that, you should use a file in the filesystem for your benchmark.

[4k]
filename=/mnt/raid10/fio.bench
rw=randrw
...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.