Veeam Monitoring ESXi CPU 102% and 103%

While using the load generator LoadGen, of DeNamiK .We got a 102% load on the physical CPU of the ESXi host according to de VEEAM Monitoring. The LoadGen is used  to simulate users activities on several Citrix virtual machines to see how many users can be active on one Citrix virtual machine and eventually on 1 ESXi host without performance degree. In the load test where we saw the 102% we had 4 virtual machines. which where configured with 2 vCPU 4 GB memory. The host is en ESXi 4.1 with the latest and greatest patches installed. The resources pool seen in the picture is not visible in vCenter but is the mother (root) resource pool of VMware. There are no “handmade” resources pools on the server.

Veeam monitor dashboard

 

And later on we got a  103% load on the esxi host

CPU usage Veeam monitor dashboard

Did i just virtual over clock my ESXi host or what?

When looking feather in to this offset i also noticed that the there is a small difference between the numbers of vCenter and Veeam monitor. Veeam takes the limit  directly from the vCenter and than takes the average usage of each virtual machine for the last 15 minutes and sums it up. That can cause a small offset.

I have made contact to Veeam support and they responded very quickly!  I never had contact with Veeam support before and i am impressed by there speed!

they investigated the logs that are under Veeam Monitor Server Settings>Logs tab, looked at setting and i have send them multiple screen shots.  Until now we did not find a answer for it yet.

I was documenting the settings that must be done for the BIOS of the ESXi host a blade server(HP Proliant BL 460c G6 ). This is where i noticed a setting i have never noticed before. This is the Intel® Turbo Boost Technology.

image 

This could be the explanation for a more than 100% CPU if the CPU speed is not dynamically update in VMware and is being used for calculation. There are no load tests planed for now. So i can not check this and i do not remember seeing a load of the CPU more than 100% in vCenter.

I talked this idea trough with support of Veeam and they will look in to this. i will update this post when i have got news.

** Update  **

Although we can not verified this at the moment  the best explanation for this behaviour is the Intel® Turbo Boost Technology. Why? Veeam monitoring get his information from vCenter and vCenter can give processor use above 100% see picture below.

vCenter above 100%

VMware makes default reservations for mother Resource Pool “Resources”. This reservation will never allow VMs to use CPU and Memory resources more than specified. For example  we have these resources available on one ESX hosts and we have

two of those in a cluster (values from VMware vSphere client):

CPU available 8x2666MHz = 21328MHz per host

CPU available in cluster =42656 MHz

Memory = 47,99 GB per host

Memory available in cluster =95,98 GB

Veeam Monitor shows me the same values for the same cluster

Also it shows me these values under root “Resources” resource pool:
CPU Limit = 37065MHz
Memory Limit = 86986 MB = 86,98 GB

Any of VMs under root resource pool consumes some resources. This leads to different CPU and Memory usage values under host and its “Resources” resource pool.
Of course, ESX host requires some resources for its daily basis operation and tasks. That is why these limitations are exist. These limitations are default limitations by VMware.

When the host has the possibility to run on a higher frequently say 3GHz  (Intel® Turbo Boost Technology) so in theory you can use a extra 334 MHz. Because the extra or higher frequency is not updated   in vCenter this will also not be seen in Veeam monitoring because they get the data from vCenter.

This is why you may get to a point where you can get more than 100%. More is not always better because when your Host is hitting a 100% or more you do have a problem and you will not care about the extra percentages.