Equallogic Automated Performance Load-balancing

About a week ago I was troubleshooting some latency issues on our Equallogic SAN, and came across a nice illustration of APLB in action. The basic test was this – run a consistent load on a LUN to see if we could reproduce the latency spikes I was intermittently picking up on our SQL instances. That way we could determine if it was a constant issue, or something to do with the load that SQL was putting on the SAN.

Our SAN is made up of the following: 1 x PS6510X and 3 x PS6010S units. All 10GbE, running on their own set of switches. We upgraded to FW5.1.1-H1 about a month ago (just before H2 was released) and we had enabled load balancing to allow unused SSD capacity to enhance our SUMO.

IOMeter Setting:

8K request size

Single worker

100% Read

100% Random

1 outstanding request (To make sure it wasn’t going to affect the rest of the SAN)

10GB test file (To make sure it wasn’t cached)

LUN on the SUMO (to begin with…)

Initially, we were seeing reasonably consistent performance and also managed to get one some of those spikes to appear.

Avg latency  - 8ms

Avg IOPS – 180

I then left this running overnight to see if the issue became more obvious over time.

In the morning, it looked like this:

Avg latency - <1ms

Avg IOPS - 1K+

What had happened is that APLB had kicked in and over time realised that some blocks in this LUN were doing more work than some others that had been migrated to SDD member, and then proceeded to move these over. By morning, the entire 10GB LUN had been moved and was running an order of magnitude faster than it was the day before.

You may be wondering why there were only 1K IOPS when this was sitting on SSD, but remember that I only had a single request outstanding, and therefore was more than likely being throttled by the network latency rather than the SSD storage itself.

To be fair, EQL SSD units are not the fastest – in fact I would go as far as to say that they are one of the slowest rackmount 10GbE SSD implementations out there. However, they are faster than any configuration of HDD’s you could fit into the same form factor, and we purchased them on the basis that they were good enough for what we wanted to do with them at the time.

So in conclusion:

·         Equallogic Load balancing across SAN members works, and does so especially well on consistent loads (The jury is still out on volatile workloads)

·         Mixing SSD and HDD models can have large benefits, if particular if there is a subset of data that you want to accelerate of a large LUN, but don’t want to put the whole LUN on SSD.

·         The load balancing happens automatically – you just turn it on for the group and it will do the rest. This is both a positive and a negative – it works but you lose control of the placement of the data. You can use volume pinning to fix a volume to a member, but that’s another story …

·         Load balancing is not instant – it will take time to kick in. It appears to me that capacity balancing kicks in first, and then performance load balancing kicks in afterwards.

My final slide deck from SQLbits 9.
http://bit.ly/qtlLyM

Thanks to all who attended - please send in your feedback on my session.

SQLBits Slide Deck

Here is my provisional slide deck for my SQLBits 9 session:
SQL Storage for the Real World:

Any feedback will be appreciated!
I will be presenting in the Aintree room, on Saturday, at 16:00 - hope to see you there!

SQLBits Prep...

An overview of what I’ll be covering in my SQLBits session

I have pretty much finished up my slides for my session at SQLBits, and now I’m going over the content I want to discuss for each slide. To give you some idea of what I’m going to discuss, here is the summary page from the session:

• Requirements
• Storage concepts
• Storage options and priorities
• SAN / DAS
• HA / DR implications
• Decide on a storage strategy
• General guidance

It feels fairly easy to talk about the storage concepts and technologies for ages on end, but the problem is that it can come across as a little dry. I did a quick run through with my team last week and I nearly put some of them to sleep. So this week and next is all about making the content interesting!


I was originally going to avoid using vendor names and naming products, but then the whole discussion becomes very theoretical and difficult to relate to. Therefore I’m going to try to be fair and balanced with everyone I discuss and not mention any products that I really don’t like...


I promise to make the session as informative and interesting as possible!

The reason for my silence is...

My submission for the SQLBits 9 conference (Query across the Mersey) has been accepted!!!
Been spending a lot of time working on my presentation, so hopefully it will go well!

I have never presented at a conference before, so I'm a little anxious but also very excited at the prospect!
My session is the last session on Saturday (community day), so if you can make it I would be most grateful!
It is titled: SQL Storage for the real world.


If it goes well, I will expand it into a series of articles here on my blog.

More info on the conference:
http://sqlbits.com/default.aspx

And this is what I was talking about...

In my previous entry I mentioned that you could build a SAN that performed better and cost less than your commercial SANs out there. Now I'm not going to make light of doing so, but here is an article to validate what I was trying to say:

http://www.theregister.co.uk/2011/09/20/nexenta_vmworld_2011/

"Cheap and cheerful storage from Nexenta matched EMC and NetApp's multi-million-dollar systems in VMworld 2011's Hands On Lab, and took up some of the slack from its rivals when difficulties arose on the first day. EMC and NetApp have cooly brushed off Nexenta's claims it outperformed them."

Really good to see some competition out there, as well as some proof that there isn't too much magic underneath the SAN covers...

Build Your Own Storage Array?

So for me it's a massive temptation to build my own storage array and pop it into production...
It's significantly cheaper and possibly more perfomant than buying a pre-configured brand name unit, and it would be fun to build. Idea's of slapping multiple PCI SSD's into a supermicro chassis, and then filling the drive cages with big SATA tubs really excites me! You can get the best of both worlds - blazing fast SSD's plus big storage all in one unit. Spec up lots of RAM for a really fat cache and you have the potential to blow away most units on the market.

You get some amazing software to give you as little or as much functionality as you could possibly want - eOpen, SAN Symphony, Starwind etc. all give you amazing features or you could go VSA and try the Lefthand or other virtual SAN solution...

Yes I know these idea's are naive, but it's the basis for a lot of the storage companies out there...
So if the hardware is there, and the software is there, then why not...?

Simple - when this solution has problems, all responsibility will lie with you alone. You can't call in the cavalry when you suddenly find there is a massive compatibility issue with a new software version or firmware edition.
I don't know about you, but I have enough stress in my job already, and don't need to be lying in bed at night wondering if a new firmware version is going to fix a bug or create new ones for me... I would rather leave that to a team of professionals who focus all their time and effort on this exact question.

Is there a time I would do it? Yes, definitely. If cost was the single biggest motivating factor for the company, or if I was unable to attain the performance levels that my system required under a specific budget. Also these kinds of solutions are great for dev and test, as long as downtime is permissible and you don't have a 5 9's SLA. You can build a very big storage solution very cost effectively in this manner and you can also build some very fast systems in this way (Tony Rogerson's blog comes to mind). But they are not enterprise class solutions - and if you consider your SQL installation to be enterprise worthy, then why risk the whole thing for a few pounds?