Fun with Storage Spaces in Windows Server 2012 R2

image

Previously I took at look at a small three disk Storage Spaces configuration on Windows 8.1 for my Plex media server build. Looking at the performance of the individual disks, I grew curious so set out to explore Spaces performance in Server 2012 R2 with one extra disk. Windows 8.1 and Server 2012 R2 are cut from the same cloth with the Server product adding performance tiering, deduplication and means for larger scale.  At its core, Storage Spaces is essentially the same between the products and accomplishes a singular goal: take several raw disks and create a pool with adjustable performance and resiliency characteristics. The execution of this however, is very different between the products.

For my test bed I used a single Dell PowerEdge R610 which has 6 local 15K SAS disks attached to a PERC H700 RAID controller with 1GB battery-backed cache. Two disks in RAID 1 for the OS (Windows Server 2012 R2) and four disks left for Spaces. Spaces is a very capable storage aggregation tool that can be used with large external JBODs and clustered across physical server nodes, but for this exercise I want to look only at local disks and compare performance between three and four disk configurations.

Questions I’m looking to answer:

  • Do 2 columns outperform 1? (To learn about columns in Spaces, read this great article)
  • Do 4-disk Spaces greatly outperform 3-disk Spaces?
  • What advantage, if any, does controller cache bring to the table?
  • Are there any other advantages in Spaces for Server 2012 R2 that outperform the Windows 8.1 variant (aside from SSD tiering, dedupe and scale)?

No RAID

The first rule of Storage Spaces is: thou shalt have no RAID in between the disks and the Windows OS. Spaces handles the striping and resiliency manipulation itself so can’t have hardware or software RAID in the way. The simplest way to accomplish this is via the use of a pass-through SAS HBA but if you only have RAID controller at your disposal, it can work by setting all disks to RAID 0. For Dell servers, in the OpenManage Server Administrator (OMSA) console, create a new Virtual Disk for every physical disk you want to include, and set its layout to RAID-0. *NOTE – There are two uses of the term “virtual disk” in this scenario. Virtual Disk in the Dell storage sense is not to be confused with virtual disk in the Windows sense, although both are ultimately used to create a logical disk from a physical resource.

image

To test the native capability of Spaces with no PERC hardware cache assist, I set the following policy on each Virtual Disk:

image

Make some Spaces!

Windows will now see these local drives as assignable disks to a Storage Pool. From the File and Storage Services tab in Server Manager, launch the New Storage Pool Wizard. This is where creating Spaces in Windows 8.1 and Server 2012 R2 divest in a major way. What took just a few clicks on a single management pane in Win8.1 takes several screens and multiple wizards in Server 2012 R2. Also, all of this can be done in PowerShell using the new-storagepool and new-virtualdisk cmdlets. First name the new storage pool:

image

Add physical disks to the pool, confirm and commit.

image

You will now see your new pool created under the Storage Spaces section. Next create new virtual disks within the pool by right-clicking and selecting “New Virtual Disk”.

image

Select the pool to create the Virtual Disk within and name it. You’ll notice that this is where you would enable performance tiering if you had the proper disks in place using at least one SDD and one HDD.

image

Choose the Storage Layout (called resiliency in Win8.1 and in PowerShell). I previously determined that Mirror Spaces are the best way to go so will be sticking with that layout for these investigations.

image

Choices are good, but we want to save committed capacity, choose Thin Provisioning:

image

Specify the size:

image

Confirm and commit:

image

Now we have a pool (Storage Space) and a Virtual Disk, now we need to create a volume. Honestly I don’t know why Microsoft didn’t combine this next step with the previous Virtual Disk creation dialogs, as a Virtual Disk with no file system or drive letter is utterly useless.

image

Next select the size of the volume you want to create (smaller than or equal to the vDisk size specified previously), assign a drive letter and file system. It is recommended to use ReFS whenever possible. Confirm and commit.

image

Finally, a usable Storage Space presented to the OS as a drive letter!

Performance Baseline

First, to set a baseline I created a 4-disk RAID10 set using the integrated PERC H700 with 1GB cache (enabled). I copied a 6GB file back and forth between the C:\ and M:\ drives to test 100% read and 100% write operations. Write IO was fairly poor not even breaking 20 IOPS, the RAID10 double write penalty in full effect. Reads were much better, as expected, nearing 300 IOPS at peak.

image

3-Disk Mirror Space Performance

To force the server to go to disk for all IO operations (since I simply moved the same 6GB file between volumes), I used a little utility called Flush Windows Cache File by DelphiTools. This very simple CLI tool flushes the file cache and memory working sets along with the modified and purged memory standby lists.

Test constants: 3-disk Mirror space, PERC caching disabled, single column (default for 3 disks), 6GB file used to simulate a 100% read operation and a 100% write operation:image

image

Here are a few write operations followed by a read operation to demonstrate the disk performance capabilities. Write IOPS spike ~30 with read IOPS ~170. Interestingly, the write performance is better than RAID10 on the PERC here with read performance at almost half.

image

Turning the PERC caching back on doesn’t help performance at all, for reads or writes.

image

4-Disk Space, 1 Column Performance

Test constants: 4-disk Mirror space, PERC caching enabled, single column, 6GB file used to simulate a 100% read operation and a 100% write operation:

When adding a 4th disk to a 3-disk Space, the column layout has already been determined to be singular, so this does not change. To use a two column layout with four disks, the space would need to be destroyed and recreated. Let’s see how four disks perform with one column.

image

You’ll notice in the vDisk output that the Allocated size is larger now with the 4th disk, but the number of columns is unchanged at 1.

image

Adding the 4th disk to the existing 3-disk Space doesn’t appear to have helped performance at all. Even with PERC caching enabled, there is no discernable difference in performance between 3 disks or 4 using a single column, so the extra spindle adds nothing here. Disabling PERC cache appears to make no difference so I’ll skip that test here.

image

4-Disk Space, 2 Column Performance

Test constants: 4-disk Mirror space, PERC caching enabled, two columns, 6GB file used to simulate a 100% read operation and a 100% write operation.

Using 4 disks by default will create Virtual Disks with 2 columns as confirmed in PowerShell, this is also the minimum number of disks required to use two columns:

image

WOW! Huge performance difference with four disks and two columns. The read performance is consistently high (over 700 IOPS) with the writes performing in a wider arc upwards of 300 IOPS. This is a 1000% performance increase for writes and 300% increase for reads. Literally off the chart! Spaces is allowing each of these participating disks to exhibit their full performance potential individually.

image

Thinking this is too good to be true, I disabled the PERC cache and ran the test again. This is no fluke, I got very close to the same results the second time with all disks participating. Incredible.

image

Plex Performance

Since I began this whole adventure with Plex in mind, fitting that I should end it there too. Streaming a compatible file across the network yielded the following results. Skipping around in the file, all four disks were read from at some point during the stream, half spiking ~110 IOPS, the other two ~80 IOPS. This is twice the amount of observed performance on my home 3-disk 7.2k SATA-based HTPC.

image

Conclusions:

Using 15K SAS disks here, which are theoretically capable of ~200 IOPS each, performance is very good overall and seems to mitigate much of the overhead typically seen in a traditional RAID configuration. Spaces provides the resilient protection of a RAID configuration while allowing each disk in the set to maximize its own performance characteristics. The 3-disk one column Spaces configuration works but is very low performing. Four disks with one column don’t perform any better, so there is significant value in running a configuration that supports two or more columns. Columns are what drive the performance differential and the cool thing about Spaces is that they are tunable. Four disks with a two column configuration should be your minimum consideration for Spaces, as the performance increase adding that second column is substantial. Starting with three disks and adding a fourth will persist your single column configuration so is not advised. Best to just start with at least four disks. PERC caching showed to neither add nor detract from performance when used in conjunction with Spaces, so you could essentially ignore it altogether. Save your money and invest in SAS pass-through modules instead. The common theme that sticks out here is that the number of columns absolutely matters when it comes to performance in Spaces. The column-to-disk correlation ratio for a two-way mirror is 1:2, so you have to have a minimum of four disks to use two columns. Three disks and one column work fine for Plex but if you can start with four, do that.

Resources:

Storage Spaces FAQs

Storage Spaces – Designing for Performance

This post originated at Exit.the.Fast.Lane | Weestro.blogspot.com

9 comments:

  1. Awesome work- that helps a lot as I'm new to Spaces and Plex.

    ReplyDelete
  2. very interesting stuff, this helped me increasing the IO speed of my server, thanks!

    ReplyDelete
  3. What's your general opinion of Storage Spaces, a good SAN alternative?

    ReplyDelete
  4. Hi, it really depends on the deployment model and use case. For Hyper-V, assuming you use scale-out file servers + JBODs and connect to shared storage via SMB, it's a great solution! Small stuff like I did here with Plex, it's great! If you have a large environment with disparate storage consumers, it probably won't replace that. But for smaller more focused architectures and deployments, it's a very good, tunable and high performance storage offering.

    ReplyDelete
  5. Currently we have a small SAN with only 2 hosts connected to it. I'm thinking about using Storage Spaces and 2-3 hosts and connecting them directly to a JBOD via SAS to take advantage of the speed of SAS. Our current hosts are connected via GigE to the SAN and I'm wondering if that's fast enough for our workload.

    ReplyDelete
  6. Oh definitely! That scenario would work fine. You can do some really cool things with spaces including SSD tiering if you need a performance boost. Watch the column layout if you go that route. Big performance upgrade for much less money.

    ReplyDelete
  7. Thanks. I spent days trying to make a parity volume work with anything better than abysmal performance, even tried adding an SSD for journaling with no help. This article convinced me to run out and go buy a 4th hard drive after 2 weekends of fiddling with raid5 (parity) and getting terrible write performance. I popped in a 4th disk and configured Raid10 (striped + mirror, 2 columns) and the performance is on par with a stripe but with the resiliency of a mirror. Storage spaces creates 2 columns for you if you choose 4 disks and a mirrored volume for your Space. Thanks again.

    ReplyDelete
  8. You bet Tim, glad you were able to achieve a successful outcome. Thanks for stopping by!

    ReplyDelete
  9. I bought 8 identical disks for a 4 column 2 way mirror in Windows 10 pro. My MB supports 10 S-ATA, plus I added a pci-express 2 port S-ATA for a second mirrored pool when I take my externals out of their cases. Any suggestions our insight for someone who has done this before?

    ReplyDelete

Powered by Blogger.