tl;dr: Make sure you flash an LSI-9211 to IT firmware rev#14 to get it to work with Linux and SSD trim. You may have to downgrade from newer firmware to older firmware to get the card to work.
( Collapse )
Finding a SATA III controller with more than one PCI-e lane
After a recent hardware issue I decided to upgrade my computer to use new Intel 520 120MB SSD drives in RAID for improved performance. The motherboard I use (an ASUS Rampage III extreme) has a Marvel SATA III controller with two ports, but I discovered that it is connected via only a single PCI-e lane (each lane can do at most 400MB/sec*). This means that it can't effectively support even a single Intel 520 because one device can saturate the SATA III bus (An Intel 520 is rated at up to 550MB/sec sequential write).
So I went on a quest for a new SATA 3 controller. To Frys! I exclaimed. But unfortunately, all the PCI-e 2.x SATA III controllers used a single lane! (These cards also feature RAID which is a double wtf).
Well, having been thwarted by my Frys run (I could have upgraded the motherboard, but I want to wait for the next generation of processors with DDR4 ram) I went to Central Computers in Mountain View. There I found LSI PCI-e HBAs. LSI HBAs use 8 PCI-e lanes. I decided to get an LSI 9211-8i for a little less than they sell on NewEgg. This is a RAID controller, and I specifically told the technician that I wanted a RAID controller that supported TRIM and he said LSI supports trim**. So I laid down my money and headed home.
Strange Windows 7 Professional 64 bit performance
I booted into Windows 7 and performance on AS-SSD (a simple benchmark utility targeted at SSD performance) for the drive was horrible. So horrible the test could not complete. During the test the performance is reported to be pegged at 98 IOPS. To compare, the same test, same drive, same OS, but different card: 13000 IOPS. 13K IOPS is low for the drive but it was an old sata 2 card. Regardless, it shows the drive can do way better than the 98 IOPS that the LSI card reported while attempting to complete the test. I messed around with this for quite awhile, pulling out hair. I eventually tried upgrading the card to R16 firmware. Same issue. I figured that maybe there was some sort of windows weirdness so I decided to test Linux. I have contacted LSI about the issue with AS-SSD to see if there is possibly a driver issue to blame.
Linux and the case of the mysterious hanging mkfs.ext4
I installed CentOS 6.4 minimal on another 32GB OCZ SSD I had lying around. CentOS found the LSI card just fine. I could run a `dd` on the Intel 520 drives and they could do 500MB/sec+ sequential write. Great. Next I tried to format a filesystem on the device using `mkfs.ext4`. ext4 now supports TRIM and will discard all of the blocks on a target when it creates a new filesystem. However, when I tried to create a filesystem with `mkfs.ext4`, it would simply hang at 0/X bytes discarded. Eventually the kernel started printing messages like "attempting task abort!" and "task abort: SUCCESS scmd" into the log.
There was no way to kill `mkfs.ext4`.
I did some searching and found an old post to the OCZ forum about problems with an OCZ Vertex 3, which is sandforce based, same as the Intel 520. When I was doing my tests, my Vertex 3 was on another SATA bus, isolated from the LSI so I think this may be SANDFORCE related.
If you didn't read the post, that user upgraded to firmware R14 to fix the problem. Unfortunately I had already upgraded to R16.
Fixing the mkfs.ext4 issue
I decided to try to downflash to R14. But of course, the flash utility won't let you downgrade by default. However, I found a workaround by using a VERY DANGEROUS mode on the controller: RESET MODE. In order to use this function you must use the DOS flash method. I created a USB FreeDOS boot disk. The flash method may work from UEFI but UEFI is a PITA. It doesn't work from the Linux flash utility or the Windows flash utility.
To place the card in RESET MODE: DO NOT REBOOT WITH THE CARD IN THIS STATE OR YOU WILL HAVE TO RMA THE CARD!!!!! C:\sas2flsh.exe -o -e 6 Now you must flash the new firmware AND bios: C:\>sas2flsh.exe -o -f FIRMWARE.bin -b BIOS.rom It is safe to reboot after flashing
With the LSI card flashed down to R14 I was able to create a filesystem on the device, but TRIM was not working. /sys/block/sda/queue/discard_max_by
It turns out only LSI HBA initiator devices support trim. So you can't use RAID and TRIM on this card or even TRIM on non-RAID devices (JBOD). Sales guy sold me another lemon. Oh well, software raid 0 isn't a large overhead.
Getting rid of raid (flash to IT mode)
Oh bugger, sas2flsh won't let you switch from IR (RAID) to IT (INITIATOR) firmware either without going into RESET mode. Follow the same instructions above, but RESET the card then flash the IT firmware instead of IR. The BIOS is the same in either case.
I'm doing software RAID1 over a portion of the two Intel, swap on a portion of the OCZ, and software RAID0 over the remainder of all three. This gives me a safe area for my OS and important files, a large high performance database test area, and the swap on the OCZ will be regarded as unused by TRIM. This will improve garbage collection on the OCZ which was in use for some time before I bought the 520s.
* You will see it quoted as 5GT/s and/or 5Gbit/sec but that is before 8b/10b encoding is applied for the data transfer over the bus
** Turns out they only support trim for HBA Initiators, not HBA RAID. You can flash to IT firmware though. See the post.
*** This post isn't MySQL specific.