Extremely slow Virtual Machines on HP Smart Array P410
I was deploying virtualized Citrix XenApp Servers on HP BL460c G6 servers and somehow the storage (direct attached) responded very slowly.
I had expected reduced performance (see my earlier post) since I didn’t have the Battery Backed Write Cache module installed.
I did order them but had to start deployment before they arrived.
I did not however expect such an extreme bad performance. Deployment took ages or sometimes failed completely and when logging in to a VM it responded very sluggish.
Disk Latency
I looked in the vSphere console what the Disk Latency was. Latency under 10ms is usually considered good while a latency between 10 and 20ms is a potential performance problem.
I was shocked to notice that the Disk Latency was much higher with peaks toward 2.000 ms (2 seconds!):
At the same time the transfer rates were very low:
CPU Ready time
A look at the CPU Ready and CPU Wait counters was the final confirmation that storage was the bottleneck.
To view these counters, select the VM in question and go to the Performance Tab and click Advanced:
Then add the counters:
These counters indicate that the CPU is waiting for something (the storage in this case).
You will want these numbers to be as low as possible and they are very high here:
Console Errors
On the VMWare console I noticed the following errors:
- scsi_cmd_alloc returned NULL!
- WARNING: NMP: nmp_DeviceRetryCommand
- WARNING: NMP: nmp_DeviceAttemptFailover
I went back from deploying multiple machines at once to just one but this didn’t improve the performance at all.
Using the error messages above I found this knowledge base article from Hewlett Packard: To Obtain Optimal Disk Subsystem Performance on ProLiant G6 Servers Configured with Smart Array P410i/P410/P411/P412/P212/P712m Controllers Running VMware.
The article states: When the Battery Backed Write Cache is not included in the configuration, even moderate disk I/O can negatively impact server performance.
In my opinion the article actually says: performance without the battery backed cache module is is absolutely horrible and there is no way you can successfully run any Virtual Machine on the hardware without it!
And then, to my great relief, the upgrade hardware (including the BBWC) was delivered. So I immediately built in the BBWC because I wanted to include the results in this post:
After placing the BBWC module it will not be activated until the battery has been sufficiently charged:
Results
And then the results are amazing:
And finally a successful deployment:
Especially the time needed for restoring the base image went down a lot (this includes not only the actual restore but also sysprep, network config and several reboots).
Related links
- To Obtain Optimal Disk Subsystem Performance on ProLiant G6 Servers Configured with Smart Array P410i/P410/P411/P412/P212/P712m Controllers Running VMware
- Slow virtual machine storage when using HP Smart Array P410 with ESX 3.5 (also on 4.1).
- Slow power on and storage operations with the HP Smart Array P410i controller
- HP 512MB P-Series Battery Backed Write Cache Upgrade
Remko Weijnen
Was once an enthusiastic PepperByte employee but is now working elsewhere. His blogs are still valuable to us and we hope to you too.