Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I recently moved from a hardware RAID1 enclosure to using two eSATA drives with md. Everything seems to be working fine, except for the fact that directory traversals/listings sometimes crawl (on the order of 10s of seconds). I am using an ext3 filesystem, with the block size set to 4K.

Here is some relevant output from commands that should be important:

mdadm --detail:

/dev/md127:
        Version : 1.2
  Creation Time : Sat Nov 16 09:46:52 2013
     Raid Level : raid1
     Array Size : 976630336 (931.39 GiB 1000.07 GB)
  Used Dev Size : 976630336 (931.39 GiB 1000.07 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Tue Nov 19 01:07:59 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Events : 19691

    Number   Major   Minor   RaidDevice State
       2       8       17        0      active sync   /dev/sdb1
       1       8        1        1      active sync   /dev/sda1

fdisk -l /dev/sd{a,b}:

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0xb410a639

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048  1953525167   976761560   83  Linux

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x261c8b44

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  1953525167   976761560   83  Linux

time dumpe2fs /dev/md127 |grep size:

dumpe2fs 1.42.7 (21-Jan-2013)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Block size:               4096
Fragment size:            4096
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal size:             128M

real    2m14.242s
user    0m2.286s
sys     0m0.352s

The way I understand it, I've got 4K sectors on these drives (recent WD reds), but the partitions/filesystems appear to be properly aligned. Since it looks like I'm using md metadata version 1.2, I think I'm also good (based on mdadm raid1 and what chunksize (or blocksize) on 4k drives?). The one thing I haven't found an answer for online is whether or not having an inode size of 256 would cause problems. Not all operations are slow, it seems that the buffer cache does a great job of keeping things zippy (as it should).

My kernel version is 3.11.2

EDIT: new info, 2013-11-19

mdadm --examine /dev/sd{a,b}1 | grep -i offset
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
    Data Offset : 262144 sectors
   Super Offset : 8 sectors

Also, I am mounting the filesystem with noatime,nodiratime I'm not really willing to mess with journaling much since if I care enough to have RAID1, it might be self-defeating. I am tempted to turn on directory indexing

EDIT 2013-11-20

Yesterday I tried turning on directory indexing for ext3 and ran e2fsck -D -f to see if that would help. Unfortunately, it hasn't. I am starting to suspect it may be a hardware issue (or is md raid1 over eSATA just really dumb to do?). I'm thinking of taking each of the drives offline and seeing how they perform alone.

EDIT 2013-11-21

iostat -kx 10 |grep -P "(sda|sdb|Device)":

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.37     1.17    0.06    0.11     1.80     5.10    84.44     0.03  165.91   64.66  221.40 100.61   1.64
sdb              13.72     1.17    2.46    0.11   110.89     5.10    90.34     0.08   32.02    6.46  628.90   9.94   2.55
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

I truncated the output past this since it was all 0.00%

I really feel like it should be irrespective of ext4 vs. ext3 because this isn't just feeling a little slower, it can take on the order of tens of seconds to a minute and some change to tab auto-complete or run an ls

EDIT: Likely a hardware issue, will close question when confirmed

The more I think of it, the more I wonder if it's my eSATA card. I'm currently using this one: http://www.amazon.com/StarTech-PEXESAT32-Express-eSATA-Controller/dp/B003GSGMPU However, I just checked dmesg and it's littered with these messages:

[363802.847117] ata1.00: status: { DRDY }
[363802.847121] ata1: hard resetting link
[363804.979044] ata2: softreset failed (SRST command error)
[363804.979047] ata2: reset failed (errno=-5), retrying in 8 secs
[363804.979059] ata1: softreset failed (SRST command error)
[363804.979064] ata1: reset failed (errno=-5), retrying in 8 secs
[363812.847047] ata1: hard resetting link
[363812.847061] ata2: hard resetting link
[363814.979063] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10)
[363814.979106] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 10)
....
[364598.751086] ata2.00: status: { DRDY }
[364598.751091] ata2: hard resetting link
[364600.883031] ata2: softreset failed (SRST command error)
[364600.883038] ata2: reset failed (errno=-5), retrying in 8 secs
[364608.751043] ata2: hard resetting link
[364610.883050] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10)
[364610.884328] ata2.00: configured for UDMA/100
[364610.884336] ata2.00: device reported invalid CHS sector 0
[364610.884342] ata2: EH complete

I am also going to buy shorter shielded eSATA cables as I'm wondering if there is some interference going on.

share|improve this question
    
just for sake of completeness regarding align, mdadm --examine /dev/sd[ab]1 should show Data Offset : 2048 sectors (or any other aligned value). –  frostschutz Nov 19 '13 at 15:44
    
I should also add that the lights on the eSATA drives never turn off, md is keeping them quite busy (though I am running things that are likely producing log files) –  zje Nov 20 '13 at 6:37
1  
Check iostat -kx 10, see if one or both of the drives is showing high %utl. Also, why are you using ext3 instead of ext4? –  derobert Nov 21 '13 at 16:24
    
Okay, will check when I get home. I'm using ext3 because I'm still (possibly unjustifiably so) concerned with the reliability of ext4. –  zje Nov 21 '13 at 21:10
1  
Yeah, those SATA resets would fully explain the slowness. As you can see, the reset chain can easily take over 10s. Your idea to switch to shorter, better cables is a good start. –  derobert Nov 22 '13 at 16:39
add comment

1 Answer

up vote 2 down vote accepted

THIS ENDED UP BEING A HARDWARE ISSUE

Switching to the new shielded cables did not help, but replacing the old card with this one: http://www.amazon.com/gp/product/B000NTM9SY did get rid of the error messages and the strange behavior. Will post something new if anything changes.

IMPORTANT NOTE FOR SATA ENCLOSURES:

Even after doing the above, any drive operation would be incredibly slow (just halt for 10-30 seconds) whenever the drive was idle for a while. The enclosure I'm using has an eSATA interface, but is powered by USB. I determined this was because it didn't have enough power to spin up, so I tried a a couple of things:

  • Using an external higher-current USB power source (in case the ports were only doing the 500mA minimum)
  • Disabling spin-down with hdparm -S 0 /dev/sdX (this alleviated the problem greatly, but did not resolve it completely)
  • Disabled advanced power management via hdparm -B 255 /dev/sdX (again, did not fully resolve)

Eventually, I discovered that Western Digital has a jumper setting for Reduced Power Spinup - designed especially for this use case!

The drives I am using are: WD Red WD10JFCX 1TB IntelliPower 2.5" http://support.wdc.com/images/kb/scrp_connect.jpg

Note that I am still operating without all the power management and spin down features (Still -B 255 and -S 0 on hdparm).

Final Verdict

Unfortunately, the RPS did not solve all of my problems, just reduced the magnitude and frequency. I believe the issues were ultimately due to the fact that the enclosure could not provide enough power (even when I use an AC-USB adapter). I eventually bought this enclosure:

http://www.amazon.com/MiniPro-eSATA-6Gbps-External-Enclosure/dp/B003XEZ33Y

and everything has been working flawlessly for the last three weeks.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.