Monday, March 25, 2013

Compare Tibco EMS Performance Under Load with Clojure

We have migrated our Tibco EMS infrastructure from the older Sun V490 servers to the newer HP ProLiant DL380 G7 servers. One of the concerns was how does the new servers perform under load compared with the older servers. I needed to create a load test that compared the performance of the two different types of servers. I turned to Clojure to help me put together this load test and here is the load test code:


Load Test Script in Clojure

; Leveraging Apache Commons to generate the random message
(def msg64 (RandomStringUtils/randomAlphanumeric 64))
(def msg128 (RandomStringUtils/randomAlphanumeric 128))
(def msg256 (RandomStringUtils/randomAlphanumeric 256))
(def msg64k (RandomStringUtils/randomAlphanumeric (* 64 1024)))
(def msg128k (RandomStringUtils/randomAlphanumeric (* 128 1024)))
(def msg256k (RandomStringUtils/randomAlphanumeric (* 256 1024)))
(def msg512k (RandomStringUtils/randomAlphanumeric (* 512 1024)))
(def msg1mb (RandomStringUtils/randomAlphanumeric (* 1024 1024)))
(def persistent-delivery (.intValue (javax.jms.DeliveryMode/PERSISTENT)))


; Send messages
(defn send-messages [server-url user password queue-name data iterations]
  (with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
            (.createQueueConnection user password))]
    (let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE) 
         queue   (.createQueue session  queue-name)]
      (with-open [sender (.createSender session queue)]
  (dotimes [_ iterations]
          (let [message (.createTextMessage session)]
            (.setJMSDeliveryMode message persistent-delivery)
            (.setText message data)
            (.send sender message)))))))
   

(def test-scenarios (list
 {:label "64 bytes" :msg msg64 :n 5000 }
 {:label "128 bytes" :msg msg128 :n 5000 }
 {:label "256 bytes" :msg msg256 :n 5000 }
 {:label "64KB bytes" :msg msg64k :n 2000 }
 {:label "128KB bytes" :msg msg128k :n 1000 }
 {:label "256KB bytes" :msg msg256k :n 1000 }
 {:label "512KB bytes" :msg msg512k :n 1000 }
 {:label "1MB bytes" :msg msg1mb :n 1000 }))

(doseq [my-test test-scenarios]
  (let [label (:label my-test)
       message (:msg my-test)
       n (:n my-test)]
    (println "\n\n")
 (println (str "Testing for " label " sized messages with n = " n))
 (time (send-messages server-url username password test-queue message n))))

Here's the message consumer part of the load test:


Message Consumer in Clojure

                
; Consume Queue Text messages asynchronously
(defn get-queue-text-messages [server-url user password queue-name process-message]
    (future
        (with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
                                   (.createQueueConnection user password))]
            (let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE)
                  queue (.createQueue session  queue-name)]
                (with-open [receiver (.createReceiver session queue)]             
                    (.start connection)
                    (loop [acc 0]                      
                        (process-message (.receive receiver) acc)
                        (recur (+ acc 1))))))))

; Dump just the message id      
(defn dump-message-id [message n]
    (println (str n)))

     
; Create function aliases with connection information embedded                   
(defn consume-messages [queue-name message-processor]
    (get-queue-text-messages  server-url username password queue-name message-processor))
 
; Start consuming messages asynchronously
(consume-messages queueName dump-message-id)     

Here are the test results:


Load Test Result on Sun V490

Message SizeNumber of MessagesProcess Time (ms)Throughput (msg/s)Throughput (MB/s)
64 5,000 2,772.84 1,803.21 0.11
128 5,000 1,943.23 2,573.04 0.31
256 5,000 1,899.24 2,632.63 0.64
64K 1,000 3,020.98 331.02 20.69
128K 1,000 4,414.00 226.55 28.32
256K 1,000 20,911.61 47.82 11.96
512K 1,000 39,213.23 25.50 12.75
1MB 1,000 80,337.55 12.45 12.45

Load Test Result on HP ProLiant DL380 G7

Message SizeNumber of MessagesProcess Time (ms)Throughput (msg/s)Throughput (MB/s)
64 5,000 1,340.22 3,730.74 0.23
128 5,000 1,345.18 3,716.99 0.45
256 5,000 1,040.59 4,804.95 1.17
64K 1,000 2,178.69 458.99 28.69
128K 1,000 3,553.02 281.45 35.18
256K 1,000 7,490.54 133.50 33.38
512K 1,000 35,078.26 28.51 14.25
1MB 1,000 77,076.51 12.97 12.97

From this test result, we can conclude that HP ProLiant DL380 G7 performed better than Sun V490 under load.

Monday, March 04, 2013

Retrieving VMware vSphere Performance Metrics Catalog with F#

vSphere collects a lot of performance metrics data on a VMware cluster. Our problem was that we cannot find a complete documentation on what are all those performance metrics data collected and what are the units of the collected metric data. We have asked VMware professional services engineer about this and his response was that VMware does not have the metrics data documented.

However, in Chapter 16 of the vSphere Web Services Programming Guide, it talked about the fact that you can retrieve the metrics catalog from vSphere itself. Although this document talks about retrieving the metrics catalog via vSphere Web Services SDK, you can also retrieve the metrics catalog with PowerCLI. Below is a F# script that retrieves vSphere metrics catalog:

#r @"C:\pkg\VMware\Infrastructure\vSphere PowerCLI\VMware.Vim.dll"

open System
open VMware.Vim
open System.Collections.Specialized

let client = new VimClient()
let service = client.Connect(serviceUrl)

// Must login to do anything - if you are getting null values, it means the session automatically timed out
client.Login(userId,password)

let printMetricCatalog () =
 // counterId max is arbitrarily set, I tried other values
 // and still produced the same number of returned metrics
    let counterIds = seq {1 .. 1000}
    let perfMgr = new PerformanceManager(client,client.ServiceContent.PerfManager)
    let metrics = perfMgr.QueryPerfCounter(counterIds |> Seq.toArray)

    let dumpAllMetricData (metric:PerfCounterInfo) = 
        printfn "-------------------------------------------------------------------\n"
        printfn "Key             : %i" metric.Key
        printfn "Level           : %A" metric.Level
        printfn "PerDeviceLevel  : %A" metric.PerDeviceLevel
        printfn "RollupType      : %A" metric.RollupType
        printfn "StatsType       : %A" metric.StatsType
        printfn "Group.Label     : %s" metric.GroupInfo.Label
        printfn "Group.Key       : %s" metric.GroupInfo.Key
        printfn "Group.Summary   : %s" metric.GroupInfo.Summary
        printfn "NameInfo.Label  : %s" metric.NameInfo.Label
        printfn "NameInfo.Key    : %s" metric.NameInfo.Key
        printfn "NameInfo.Summary: %s" metric.NameInfo.Summary
        printfn "UnitInfo.Label  : %s" metric.UnitInfo.Label
        printfn "UnitInfo.Key    : %s" metric.UnitInfo.Key
        printfn "UnitInfo.Summary: %s" metric.UnitInfo.Summary


    let dumpMetricData (metric:PerfCounterInfo) = printfn "\t%s (%s) - %A stat,  rollup: %A" 
                                                          (metric.NameInfo.Label) 
                                                          (metric.UnitInfo.Label) 
                                                          (metric.StatsType)
                                                          (metric.RollupType)

    // Break metrics catalog by metric groups
    let metricGroups = metrics |> Seq.map (fun metric -> metric.GroupInfo.Label) |> Set.ofSeq |> Set.toSeq

    // Dump short version of metrics catalog
    metricGroups
    |> Seq.iter (fun group ->  printfn "\n--------------------------------------------------"
                               printfn " Metric Group : %s" group
                               printfn "--------------------------------------------------"
                               metrics
                               |> Seq.filter (fun metric -> metric.GroupInfo.Label = group)
                               |> Seq.iter dumpMetricData)

    // Dump complete version of metrics catalog
    metrics |> Seq.iter dumpAllMetricData

printMetricCatalog()

Here's a dump of the metrics catalog reformatted to display as HTML tables:

VMware vSphere Metrics Catalog


CPU

MetricUnitStats TypeRollupDescription
CPU Capacity Contention Percent rate average Percent of time the VMs on this host are unable to run because they are contending for access to the physical CPU(s)
CPU Capacity Demand MHz absolute average The amount of CPU resources VMs on this host would use if there were no CPU contention or CPU limit
CPU Capacity Entitlement MHz absolute average CPU resources devoted by the ESX scheduler to virtual machines and resource pools
CPU Capacity Provisioned MHz absolute average Capacity in MHz of the physical CPU cores
CPU Capacity Usage MHz rate average CPU usage in MHz during the interval
CPU Core Count Contention Percent rate average Time the VM is ready to run, but is unable to run due to co-scheduling constraints
CPU Core Count Provisioned Number absolute average The number of physical cores provisioned to the entity
CPU Core Count Usage Number absolute average The number of virtual processors running on the host
Co-stop Millisecond delta summation Time the VM is ready to run, but is unable to due to co-scheduling constraints
Core Utilization Percent rate none CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Core Utilization Percent rate average CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Core Utilization Percent rate maximum CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Core Utilization Percent rate minimum CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Demand MHz absolute average The amount of CPU resources a VM would use if there were no CPU contention or CPU limit
Entitlement MHz absolute latest CPU resources devoted by the ESX scheduler
Idle Millisecond delta summation Total time that the CPU spent in an idle state
Latency Percent rate average Percent of time the VM is unable to run because it is contending for access to the physical CPU(s)
Max limited Millisecond delta summation Time the VM is ready to run, but is not run due to maxing out its CPU limit setting
Overlap Millisecond delta summation Time the VM was interrupted to perform system services on behalf of that VM or other VMs
Ready Millisecond delta summation Percentage of time that the virtual machine was ready, but could not get scheduled to run on the physical CPU
Reserved capacity MHz absolute average Total CPU capacity reserved by virtual machines
Run Millisecond delta summation Time the VM is scheduled to run
Swap wait Millisecond delta summation CPU time spent waiting for swap-in
System Millisecond delta summation Amount of time spent on system processes on each virtual CPU in the virtual machine
Total MHz rate average Total amount of CPU resources of all hosts in the cluster
Total capacity MHz absolute average Total CPU capacity reserved by and available for virtual machines
Usage Percent rate none CPU usage as a percentage during the interval
Usage Percent rate average CPU usage as a percentage during the interval
Usage Percent rate minimum CPU usage as a percentage during the interval
Usage Percent rate maximum CPU usage as a percentage during the interval
Usage in MHz MHz rate none CPU usage in megahertz during the interval
Usage in MHz MHz rate average CPU usage in megahertz during the interval
Usage in MHz MHz rate minimum CPU usage in megahertz during the interval
Usage in MHz MHz rate maximum CPU usage in megahertz during the interval
Used Millisecond delta summation Total CPU usage
Utilization Percent rate none CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Utilization Percent rate average CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Utilization Percent rate maximum CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Utilization Percent rate minimum CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Wait Millisecond delta summation Total CPU time spent in wait state
Worst case allocation MHz absolute latest Amount of CPU resources allocated to the virtual machine or resource pool based on the total cluster capacity and the resource configuration of the resource hierarchy

Cluster services

MetricUnitStats TypeRollupDescription
CPU fairness Number absolute latest Fairness of distributed CPU resource allocation
Current failover level Number absolute latest vSphere HA number of failures that can be tolerated
Effective CPU resources MHz rate average Total available CPU resources of all hosts within a cluster
Effective memory resources MB absolute average Total amount of machine memory of all hosts in the cluster that is available for use for virtual machine memory and overhead memory
Memory fairness Number absolute latest Aggregate available memory resources of all the hosts within a cluster

Datastore

MetricUnitStats TypeRollupDescription
Average read requests per second Number rate average Average number of read commands issued per second to the datastore during the collection interval
Average write requests per second Number rate average Average number of write commands issued per second to the datastore during the collection interval
Highest latency Millisecond absolute latest Highest latency value across all datastores used by the host
Read latency Millisecond absolute average The average time a read from the datastore takes
Read rate KBps rate average Rate of reading data from the datastore
Storage DRS datastore bytes read Number absolute latest Storage DRS datastore bytes read
Storage DRS datastore bytes written Number absolute latest Storage DRS datastore bytes written
Storage DRS datastore normalized read latency Number absolute latest Storage DRS datastore normalized read latency
Storage DRS datastore normalized write latency Number absolute latest Storage DRS datastore normalized write latency
Storage DRS datastore outstanding read requests Number absolute latest Storage DRS datastore outstanding read requests
Storage DRS datastore outstanding write requests Number absolute latest Storage DRS datastore outstanding write requests
Storage DRS datastore read I/O rate Number absolute latest Storage DRS datastore read I/O rate
Storage DRS datastore read workload metric Number absolute latest Storage DRS datastore metric for read workload model
Storage DRS datastore write I/O rate Number absolute latest Storage DRS datastore write I/O rate
Storage DRS datastore write workload metric Number absolute latest Storage DRS datastore metric for write workload model
Storage I/O Control aggregated IOPS Number absolute average Storage I/O Control aggregated IOPS
Storage I/O Control datastore maximum queue depth Number absolute latest Storage I/O Control datastore maximum queue depth
Storage I/O Control normalized latency Microsecond absolute average Storage I/O Control size-normalized I/O latency
Write latency Millisecond absolute average The average time a write to the datastore takes
Write rate KBps rate average Rate of writing data to the datastore
busResets Number delta summation busResets
commandsAborted Number delta summation commandsAborted
contention Millisecond absolute average contention
usage KBps absolute average usage

Disk

MetricUnitStats TypeRollupDescription
Average commands issued per second Number rate average Average number of SCSI commands issued per second during the collection interval
Average read requests per second Number rate average Average number of disk reads per second during the collection interval
Average write requests per second Number rate average Average number of disk writes per second during the collection interval
Bus resets Number delta summation Number of SCSI-bus reset commands issued during the collection interval
Capacity KB absolute latest Configured size of the datastore
Command latency Millisecond absolute average Average amount of time taken during the collection interval to process a SCSI command issued by the Guest OS to the virtual machine
Commands issued Number delta summation Number of SCSI commands issued during the collection interval
Commands terminated Number delta summation Number of SCSI commands terminated during the collection interval
Disk SCSI Reservation Conflicts Number delta summation Number of SCSI reservation conflicts for the LUN during the collection interval
Disk SCSI Reservation Conflicts % Percent absolute average Number of SCSI reservation conflicts for the LUN as a percent of total commands during the collection interval
Disk Throughput Contention Millisecond absolute average Average amount of time for an I/O operation to complete
Disk Throughput Usage KBps rate average Aggregated disk I/O rate
Highest latency Millisecond absolute latest Highest latency value across all disks used by the host
Kernel command latency Millisecond absolute average Average amount of time, in milliseconds, spent by VMkernel to process each SCSI command
Kernel read latency Millisecond absolute average Average amount of time, in milliseconds, spent by VMKernel to process each SCSI read command
Kernel write latency Millisecond absolute average Average amount of time, in milliseconds, spent by VMKernel to process each SCSI write command
Maximum queue depth Number absolute average Maximum queue depth
Overhead due to delta disk backings KB absolute latest Storage overhead of a virtual machine or a datastore due to delta disk backings
Physical device command latency Millisecond absolute average Average amount of time, in milliseconds, to complete a SCSI command from the physical device
Physical device read latency Millisecond absolute average Average amount of time, in milliseconds, to read from the physical device
Physical device write latency Millisecond absolute average Average amount of time, in milliseconds, to write to the physical device
Queue command latency Millisecond absolute average Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval
Queue read latency Millisecond absolute average Average amount of time spent in the VMkernel queue, per SCSI read command, during the collection interval
Queue write latency Millisecond absolute average Average amount time spent in the VMkernel queue, per SCSI write command, during the collection interval
Read latency Millisecond absolute average Average amount of time taken during the collection interval to process a SCSI read command issued from the Guest OS to the virtual machine
Read rate KBps rate average Average number of kilobytes read from the disk each second during the collection interval
Read requests Number delta summation Number of disk reads during the collection interval
Space actually used KB absolute latest Amount of space actually used by the virtual machine or the datastore
Space not shared KB absolute latest Amount of space associated exclusively with a virtual machine
Space potentially used KB absolute latest Amount of storage set aside for use by a datastore or a virtual machine
Usage KBps rate none Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Usage KBps rate average Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Usage KBps rate minimum Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Usage KBps rate maximum Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Write latency Millisecond absolute average Average amount of time taken during the collection interval to process a SCSI write command issued by the Guest OS to the virtual machine
Write rate KBps rate average Average number of kilobytes written to disk each second during the collection interval
Write requests Number delta summation Number of disk writes during the collection interval
contention Percent absolute average contention
provisioned KB absolute average provisioned
usage KB absolute average usage

Management agent

MetricUnitStats TypeRollupDescription
CPU usage MHz rate average Amount of Service Console CPU usage
Memory swap in KBps rate average Amount of memory that is swapped in for the Service Console
Memory swap out KBps rate average Amount of memory that is swapped out for the Service Console
Memory swap used KB absolute average Sum of the memory swapped by all powered-on virtual machines on the host
Memory used KB absolute average Amount of total configured memory that is available for use

Memory

MetricUnitStats TypeRollupDescription
Active KB absolute none Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active KB absolute average Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active KB absolute minimum Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active KB absolute maximum Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active write KB absolute average Amount of memory that is actively being written to by the VM
Balloon KB absolute none Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon KB absolute average Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon KB absolute minimum Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon KB absolute maximum Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon target KB absolute none Target value set by VMkernal for the virtual machine's memory balloon size
Balloon target KB absolute average Target value set by VMkernal for the virtual machine's memory balloon size
Balloon target KB absolute minimum Target value set by VMkernal for the virtual machine's memory balloon size
Balloon target KB absolute maximum Target value set by VMkernal for the virtual machine's memory balloon size
Compressed KB absolute average Amount of memory compressed by ESX
Compression rate KBps rate average Rate of memory compression for the VM
Consumed KB absolute none Amount of memory consumed by a virtual machine, host, or cluster
Consumed KB absolute average Amount of memory consumed by a virtual machine, host, or cluster
Consumed KB absolute minimum Amount of memory consumed by a virtual machine, host, or cluster
Consumed KB absolute maximum Amount of memory consumed by a virtual machine, host, or cluster
Decompression rate KBps rate average Rate of memory decompression for the VM
Entitlement KB absolute average Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
Granted KB absolute none Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Granted KB absolute average Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Granted KB absolute minimum Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Granted KB absolute maximum Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Heap KB absolute none VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap KB absolute average VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap KB absolute minimum VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap KB absolute maximum VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap free KB absolute none Free address space in the VMkernel's main heap
Heap free KB absolute average Free address space in the VMkernel's main heap
Heap free KB absolute minimum Free address space in the VMkernel's main heap
Heap free KB absolute maximum Free address space in the VMkernel's main heap
Host cache used for swapping KB absolute none Space used for caching swapped pages in the host cache
Host cache used for swapping KB absolute average Space used for caching swapped pages in the host cache
Host cache used for swapping KB absolute maximum Space used for caching swapped pages in the host cache
Host cache used for swapping KB absolute minimum Space used for caching swapped pages in the host cache
Latency Percent rate average Percentage of time the VM is waiting to access swapped or compressed memory
Low free threshold KB absolute average Threshold of free host physical memory below which ESX will begin reclaiming memory from VMs through ballooning and swapping
Memory Capacity Contention Percent rate average Percentage of time the VM is waiting to access swapped, compressed, or ballooned memory
Memory Capacity Entitlement KB absolute average Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
Memory Capacity Provisioned KB absolute average Total amount of memory configured for the VM
Memory Capacity Usable KB absolute average Amount of physical memory available for use by virtual machines on this host
Memory Capacity Usage KB absolute average Amount of physical memory actively used
Memory Consumed by VMs KB absolute average Amount of physical memory consumed by VMs on this host
Memory Consumed by userworlds KB absolute average Amount of physical memory consumed by userworlds on this host
Memory Reserved Capacity % Percent absolute average Percent of memory that has been reserved either through VMkernel use, by userworlds, or due to VM memory reservations
Memory saved by zipping KB absolute latest Memory (KB) saved due to memory zipping
Overhead KB absolute none Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead KB absolute average Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead KB absolute minimum Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead KB absolute maximum Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead touched KB absolute average Actively touched overhead memory (KB) reserved for use as the virtualization overhead for the VM
Reserved capacity MB absolute average Total amount of memory reservation used by powered-on virtual machines and vSphere services on the host
Reserved overhead KB absolute average Memory (KB) reserved for use as the virtualization overhead for the VM
Shared KB absolute none Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared KB absolute average Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared KB absolute minimum Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared KB absolute maximum Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared common KB absolute none Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
Shared common KB absolute average Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
Shared common KB absolute minimum Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
Shared common KB absolute maximum Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
State Number absolute latest One of four threshold levels representing the percentage of free memory on the host. The counter value determines swapping and ballooning behavior for memory reclamation.
Swap in KB absolute none Amount swapped-in to memory from disk
Swap in KB absolute average Amount swapped-in to memory from disk
Swap in KB absolute maximum Amount swapped-in to memory from disk
Swap in KB absolute minimum Amount swapped-in to memory from disk
Swap in from host cache KB absolute none Amount of memory swapped-in from host cache
Swap in from host cache KB absolute average Amount of memory swapped-in from host cache
Swap in from host cache KB absolute maximum Amount of memory swapped-in from host cache
Swap in from host cache KB absolute minimum Amount of memory swapped-in from host cache
Swap in rate KBps rate average Rate at which memory is swapped from disk into active memory during the interval
Swap in rate from host cache KBps rate average Rate at which memory is being swapped from host cache into active memory
Swap out KB absolute none Amount of memory swapped-out to disk
Swap out KB absolute average Amount of memory swapped-out to disk
Swap out KB absolute maximum Amount of memory swapped-out to disk
Swap out KB absolute minimum Amount of memory swapped-out to disk
Swap out rate KBps rate average Rate at which memory is being swapped from active memory to disk during the current interval
Swap out rate to host cache KBps rate average Rate at which memory is being swapped from active memory to host cache
Swap out to host cache KB absolute none Amount of memory swapped-out to host cache
Swap out to host cache KB absolute average Amount of memory swapped-out to host cache
Swap out to host cache KB absolute maximum Amount of memory swapped-out to host cache
Swap out to host cache KB absolute minimum Amount of memory swapped-out to host cache
Swap target KB absolute none Target size for the virtual machine swap file
Swap target KB absolute average Target size for the virtual machine swap file
Swap target KB absolute minimum Target size for the virtual machine swap file
Swap target KB absolute maximum Target size for the virtual machine swap file
Swap unreserved KB absolute none Amount of memory that is unreserved by swap
Swap unreserved KB absolute average Amount of memory that is unreserved by swap
Swap unreserved KB absolute minimum Amount of memory that is unreserved by swap
Swap unreserved KB absolute maximum Amount of memory that is unreserved by swap
Swap used KB absolute none Amount of memory that is used by swap
Swap used KB absolute average Amount of memory that is used by swap
Swap used KB absolute minimum Amount of memory that is used by swap
Swap used KB absolute maximum Amount of memory that is used by swap
Swapped KB absolute none Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Swapped KB absolute average Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Swapped KB absolute minimum Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Swapped KB absolute maximum Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Total MB absolute average Total amount of machine memory of all hosts in the cluster that is available for virtual machine memory (physical memory for use by the Guest OS) and virtual machine overhead memory
Total capacity MB absolute average Total amount of memory reservation used by and available for powered-on virtual machines and vSphere services on the host
Unreserved KB absolute none Amount of memory that is unreserved
Unreserved KB absolute average Amount of memory that is unreserved
Unreserved KB absolute minimum Amount of memory that is unreserved
Unreserved KB absolute maximum Amount of memory that is unreserved
Usage Percent absolute none Memory usage as percentage of total configured or available memory
Usage Percent absolute average Memory usage as percentage of total configured or available memory
Usage Percent absolute minimum Memory usage as percentage of total configured or available memory
Usage Percent absolute maximum Memory usage as percentage of total configured or available memory
Used by VMkernel KB absolute none Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Used by VMkernel KB absolute average Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Used by VMkernel KB absolute maximum Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Used by VMkernel KB absolute minimum Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Worst case allocation MB absolute latest Memory allocation as calculated by the VMkernel scheduler based on current estimated demand and reservation, limit, and shares policies set for all virtual machines and resource pools in the host or cluster
Zero KB absolute none Memory that contains 0s only
Zero KB absolute average Memory that contains 0s only
Zero KB absolute minimum Memory that contains 0s only
Zero KB absolute maximum Memory that contains 0s only
Zipped memory KB absolute latest Memory (KB) zipped
swapIn KB absolute none swapIn
swapIn KB absolute average swapIn
swapIn KB absolute minimum swapIn
swapIn KB absolute maximum swapIn
swapOut KB absolute none swapOut
swapOut KB absolute average swapOut
swapOut KB absolute minimum swapOut
swapOut KB absolute maximum swapOut
userworld KB absolute average userworld
userworld KB absolute average userworld
vm KB absolute average vm
vm KB absolute average vm
vmOvhd KB absolute average vmOvhd
vmOvrhd KB absolute average vmOvrhd
vmkOvrhd KB absolute average vmkOvrhd
vmkOvrhd KB absolute average vmkOvrhd

Network

MetricUnitStats TypeRollupDescription
Broadcast receives Number delta summation Number of broadcast packets received during the sampling interval
Broadcast transmits Number delta summation Number of broadcast packets transmitted during the sampling interval
Data receive rate KBps rate average Average rate at which data was received during the interval
Data receive rate KBps rate average Average amount of data received per second
Data transmit rate KBps rate average Average rate at which data was transmitted during the interval
Data transmit rate KBps rate average Average amount of data transmitted per second
Multicast receives Number delta summation Number of multicast packets received during the sampling interval
Multicast transmits Number delta summation Number of multicast packets transmitted during the sampling interval
Packet receive errors Number delta summation Number of packets with errors received during the sampling interval
Packet transmit errors Number delta summation Number of packets with errors transmitted during the sampling interval
Packets received Number delta summation Number of packets received during the interval
Packets transmitted Number delta summation Number of packets transmitted during the interval
Receive packets dropped Number delta summation Number of receives dropped
Transmit packets dropped Number delta summation Number of transmits dropped
Unknown protocol frames Number delta summation Number of frames with unknown protocol received during the sampling interval
Usage KBps rate none Network utilization (combined transmit-rates and receive-rates) during the interval
Usage KBps rate average Network utilization (combined transmit-rates and receive-rates) during the interval
Usage KBps rate minimum Network utilization (combined transmit-rates and receive-rates) during the interval
Usage KBps rate maximum Network utilization (combined transmit-rates and receive-rates) during the interval
pNic Packets Received and Transmitted per Second Number rate average Average rate of packets received and transmitted per second
pNic Throughput Provisioned KBps absolute average Provisioned pNic I/O Throughput
pNic Throughput Usable KBps absolute average Usable pNic I/O Throughput
pNic Throughput Usage for FT KBps rate average Average pNic I/O rate for FT
pNic Throughput Usage for NFS KBps rate average Average pNic I/O rate for NFS
pNic Throughput Usage for VMs KBps rate average Average pNic I/O rate for VMs
pNic Throughput Usage for VR KBps rate average Average pNic I/O rate for VR
pNic Throughput Usage for iSCSI KBps rate average Average pNic I/O rate for iSCSI
pNic Throughput Usage for vMotion KBps rate average Average pNic I/O rate for vMotion
vNic Throughput Contention Number delta summation Count of vNic packet drops
vNic Throughput Usage KBps rate average Average vNic I/O rate

Power

MetricUnitStats TypeRollupDescription
Cap Watt absolute average Maximum allowed power usage
Energy usage Joule delta summation Total energy used since last stats reset
Host Power Capacity Provisioned Percent absolute average Current power usage as a percentage of maximum allowed power
Host Power Capacity Usable Watt absolute average Current maximum allowed power usage
Power Capacity Usage Watt absolute average Current power usage
Usage Watt rate average Current power usage

Resource group CPU

MetricUnitStats TypeRollupDescription
Active (1 min. average) Percent absolute latest CPU active average over 1 minute
Active (1 min. peak) Percent absolute latest CPU active peak over 1 minute
Active (15 min. average) Percent absolute latest CPU active average over 15 minutes
Active (15 min. peak) Percent absolute latest CPU active peak over 15 minutes
Active (5 min. average) Percent absolute latest CPU active average over 5 minutes
Active (5 min. peak) Percent absolute latest CPU active peak over 5 minutes
Group CPU sample count Number absolute latest Group CPU sample count
Group CPU sample period Millisecond absolute latest Group CPU sample period
Running (1 min. average) Percent absolute latest CPU running average over 1 minute
Running (1 min. peak) Percent absolute latest CPU running peak over 1 minute
Running (15 min. average) Percent absolute latest CPU running average over 15 minutes
Running (15 min. peak) Percent absolute latest CPU running peak over 15 minutes
Running (5 min. average) Percent absolute latest CPU running average over 5 minutes
Running (5 min. peak) Percent absolute latest CPU running peak over 5 minutes
Throttled (1 min. average) Percent absolute latest Amount of CPU resources over the limit that were refused, average over 1 minute
Throttled (15 min. average) Percent absolute latest Amount of CPU resources over the limit that were refused, average over 15 minutes
Throttled (5 min. average) Percent absolute latest Amount of CPU resources over the limit that were refused, average over 5 minutes

Storage adapter

MetricUnitStats TypeRollupDescription
Average commands issued per second Number rate average Average number of commands issued per second by the storage adapter during the collection interval
Average read requests per second Number rate average Average number of read commands issued per second by the storage adapter during the collection interval
Average write requests per second Number rate average Average number of write commands issued per second by the storage adapter during the collection interval
Highest latency Millisecond absolute latest Highest latency value across all storage adapters used by the host
Read latency Millisecond absolute average The average time a read by the storage adapter takes
Read rate KBps rate average Rate of reading data by the storage adapter
Storage Adapter Number Queued Number absolute average The current number of I/Os that are waiting to be issued
Storage Adapter Outstanding I/Os Percent absolute average The percent of I/Os that have been issued but have not yet completed
Storage Adapter Outstanding I/Os Number absolute average The number of I/Os that have been issued but have not yet completed
Storage Adapter Queue Command Latency Millisecond absolute average Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval
Storage Adapter Queue Depth Number absolute average The maximum number of I/Os that can be outstanding at a given time
Storage Adapter Throughput Contention Millisecond absolute average Average amount of time for an I/O operation to complete
Storage Adapter Throughput Usage KBps rate average The storage adapter's I/O rate
Write latency Millisecond absolute average The average time a write by the storage adapter takes
Write rate KBps rate average Rate of writing data by the storage adapter

Storage path

MetricUnitStats TypeRollupDescription
Average commands issued per second Number rate average Average number of commands issued per second on the storage path during the collection interval
Average read requests per second Number rate average Average number of read commands issued per second on the storage path during the collection interval
Average write requests per second Number rate average Average number of write commands issued per second on the storage path during the collection interval
Highest latency Millisecond absolute latest Highest latency value across all storage paths used by the host
Read latency Millisecond absolute average The average time a read issued on the storage path takes
Read rate KBps rate average Rate of reading data on the storage path
Storage Path Bus Resets Number delta summation Number of SCSI-bus reset commands issued during the collection interval
Storage Path Command Aborts Number delta summation Number of SCSI commands aborted during the collection interval
Storage Path Throughput Contention Millisecond absolute average Average amount of time for an I/O operation to complete
Storage Path Throughput Usage KBps rate average Storage path I/O rate
Write latency Millisecond absolute average The average time a write issued on the storage path takes
Write rate KBps rate average Rate of writing data on the storage path

System

MetricUnitStats TypeRollupDescription
Disk usage Percent absolute latest Amount of disk space usage for each mount point
Heartbeat Number delta summation Number of heartbeats issued per virtual machine during the interval
Heartbeat Number absolute latest Number of heartbeats issued per virtual machine during the interval
OS Uptime Second absolute latest Total time elapsed, in seconds, since last operating system boot-up
Resource CPU active (1 min. average) Percent absolute latest CPU active average over 1 minute of the system resource group
Resource CPU active (5 min. average) Percent absolute latest CPU active average over 5 minutes of the system resource group
Resource CPU allocation maximum, in MHz MHz absolute latest CPU allocation limit, in MHz, of the system resource group
Resource CPU allocation minimum, in MHz MHz absolute latest CPU allocation reservation, in MHz, of the system resource group
Resource CPU allocation shares Number absolute latest CPU allocation shares of the system resource group
Resource CPU maximum limited (1 min.) Percent absolute latest CPU maximum limited over 1 minute of the system resource group
Resource CPU maximum limited (5 min.) Percent absolute latest CPU maximum limited over 5 minutes of the system resource group
Resource CPU running (1 min. average) Percent absolute latest CPU running average over 1 minute of the system resource group
Resource CPU running (5 min. average) Percent absolute latest CPU running average over 5 minutes of the system resource group
Resource CPU usage (Average) MHz rate average Amount of CPU used by the Service Console and other applications during the interval
Resource CPU usage (Maximum) MHz rate maximum Amount of CPU used by the Service Console and other applications during the interval
Resource CPU usage (Minimum) MHz rate minimum Amount of CPU used by the Service Console and other applications during the interval
Resource CPU usage (None) MHz rate none Amount of CPU used by the Service Console and other applications during the interval
Resource memory allocation maximum, in KB KB absolute latest Memory allocation limit, in KB, of the system resource group
Resource memory allocation minimum, in KB KB absolute latest Memory allocation reservation, in KB, of the system resource group
Resource memory allocation shares Number absolute latest Memory allocation shares of the system resource group
Resource memory mapped KB absolute latest Memory mapped by the system resource group
Resource memory overhead KB absolute latest Overhead memory consumed by the system resource group
Resource memory share saved KB absolute latest Memory saved due to sharing by the system resource group
Resource memory shared KB absolute latest Memory shared by the system resource group
Resource memory swapped KB absolute latest Memory swapped out by the system resource group
Resource memory touched KB absolute latest Memory touched by the system resource group
Resource memory zero KB absolute latest Zero filled memory used by the system resource group
Uptime Second absolute latest Total time elapsed, in seconds, since last system startup

Virtual disk

MetricUnitStats TypeRollupDescription
Average number of outstanding read requests Number absolute latest Average number of outstanding read requests to the virtual disk during the collection interval
Average number of outstanding write requests Number absolute latest Average number of outstanding write requests to the virtual disk during the collection interval
Average read requests per second Number rate average Average number of read commands issued per second to the virtual disk during the collection interval
Average write requests per second Number rate average Average number of write commands issued per second to the virtual disk during the collection interval
Read latency Millisecond absolute average The average time a read from the virtual disk takes
Read rate KBps rate average Rate of reading data from the virtual disk
Read workload metric Number absolute latest Storage DRS virtual disk metric for the read workload model
Virtual Disk Throughput Contention Millisecond absolute average Average amount of time for an I/O operation to complete
Virtual Disk Throughput Usage KBps rate average Virtual disk I/O rate
Write latency Millisecond absolute average The average time a write to the virtual disk takes
Write rate KBps rate average Rate of writing data to the virtual disk
Write workload metric Number absolute latest Storage DRS virtual disk metric for the write workload model
busResets Number delta summation busResets
commandsAborted Number delta summation commandsAborted

Virtual machine operations

MetricUnitStats TypeRollupDescription
Storage vMotion count Number absolute latest Number of migrations with Storage vMotion (datastore change operations for powered-on VMs)
VM clone count Number absolute latest Number of virtual machine clone operations
VM create count Number absolute latest Number of virtual machine create operations
VM datastore change count (non-powered-on VMs) Number absolute latest Number of datastore change operations for powered-off and suspended virtual machines
VM delete count Number absolute latest Number of virtual machine delete operations
VM guest reboot count Number absolute latest Number of virtual machine guest reboot operations
VM guest shutdown count Number absolute latest Number of virtual machine guest shutdown operations
VM host and datastore change count (non-powered-on VMs) Number absolute latest Number of host and datastore change operations for powered-off and suspended virtual machines
VM host change count (non-powered-on VMs) Number absolute latest Number of host change operations for powered-off and suspended VMs
VM power off count Number absolute latest Number of virtual machine power off operations
VM power on count Number absolute latest Number of virtual machine power on operations
VM reconfigure count Number absolute latest Number of virtual machine reconfigure operations
VM register count Number absolute latest Number of virtual machine register operations
VM reset count Number absolute latest Number of virtual machine reset operations
VM standby guest count Number absolute latest Number of virtual machine standby guest operations
VM suspend count Number absolute latest Number of virtual machine suspend operations
VM template deploy count Number absolute latest Number of virtual machine template deploy operations
VM unregister count Number absolute latest Number of virtual machine unregister operations
vMotion count Number absolute latest Number of migrations with vMotion (host change operations for powered-on VMs)

vCenter debugging information

MetricUnitStats TypeRollupDescription
Activation count Number absolute maximum Activation operations in vCenter
Activation count Number absolute minimum Activation operations in vCenter
Activation count Number absolute summation Activation operations in vCenter
Activation latency Millisecond absolute maximum The latency of an activation operation in vCenter
Activation latency Millisecond absolute minimum The latency of an activation operation in vCenter
Activation latency Millisecond absolute summation The latency of an activation operation in vCenter
Host sync count Number absolute maximum The number of host sync operations in vCenter
Host sync count Number absolute minimum The number of host sync operations in vCenter
Host sync count Number absolute summation The number of host sync operations in vCenter
Host sync latency Millisecond absolute maximum The latency of a host sync operation in vCenter
Host sync latency Millisecond absolute minimum The latency of a host sync operation in vCenter
Host sync latency Millisecond absolute summation The latency of a host sync operation in vCenter
Inventory statistics Number absolute maximum vCenter inventory statistics
Inventory statistics Number absolute minimum vCenter inventory statistics
Inventory statistics Number absolute summation vCenter inventory statistics
Locking statistics Number absolute maximum vCenter locking statistics
Locking statistics Number absolute minimum vCenter locking statistics
Locking statistics Number absolute summation vCenter locking statistics
Managed object reference statistics Number absolute maximum Managed object reference counts in vCenter
Managed object reference statistics Number absolute minimum Managed object reference counts in vCenter
Managed object reference statistics Number absolute summation Managed object reference counts in vCenter
Miscellaneous Number absolute maximum Miscellaneous statistics
Miscellaneous Number absolute minimum Miscellaneous statistics
Miscellaneous Number absolute summation Miscellaneous statistics
Scoreboard statistics Number absolute maximum Object counts in vCenter
Scoreboard statistics Number absolute minimum Object counts in vCenter
Scoreboard statistics Number absolute summation Object counts in vCenter
Session statistics Number absolute maximum The statistics of client sessions connected to vCenter
Session statistics Number absolute minimum The statistics of client sessions connected to vCenter
Session statistics Number absolute summation The statistics of client sessions connected to vCenter
System statistics Number absolute maximum The statistics of vCenter as a running system such as thread statistics and heap statistics
System statistics Number absolute minimum The statistics of vCenter as a running system such as thread statistics and heap statistics
System statistics Number absolute summation The statistics of vCenter as a running system such as thread statistics and heap statistics
vCenter LRO statistics Number absolute maximum vCenter LRO statistics
vCenter LRO statistics Number absolute minimum vCenter LRO statistics
vCenter LRO statistics Number absolute summation vCenter LRO statistics
vCenter service statistics Number absolute maximum vCenter service statistics such as events, alarms, and tasks
vCenter service statistics Number absolute minimum vCenter service statistics such as events, alarms, and tasks
vCenter service statistics Number absolute summation vCenter service statistics such as events, alarms, and tasks

vCenter resource usage information

MetricUnitStats TypeRollupDescription
CPU privileged Percent rate average CPU used by vCenter in privileged mode
CPU process Percent rate average Total CPU used by vCenter
CPU queue length Number absolute average Processor queue length on the system where vCenter is running
CPU system Percent rate average Total system CPU used on the system where vCenter in running
CPU user Percent rate average CPU used by vCenter in user mode
Context switch rate Number rate average Number of context switches per second on the system where vCenter is running
Disk bytes read rate Number rate average Number of bytes read from the disk per second on the system where vCenter is running
Disk bytes written rate Number rate average Number of bytes written to the disk per second on the system where vCenter is running
Disk queue length Number absolute average Disk queue length on the system where vCenter is running
Disk read rate Number rate average Number of disk reads per second on the system where vCenter is running
Disk write rate Number rate average Number of disk writes per second on the system where vCenter is running
Network queue length Number absolute average Network queue length on the system where vCenter is running
Network usage Percent rate average Total network bytes received and sent per second on the system where vCenter is running
Page fault rate Number rate average Number of page faults per second on the system where vCenter is running
Physical memory KB absolute average Physical memory used by vCenter
Pool non-paged bytes KB absolute average Memory pooled for non-paged bytes on the system where vCenter is running
Pool paged bytes KB absolute average Memory pooled for paged bytes on the system where vCenter is running
Process handles Number absolute average Handles used by vCenter
Process threads Number absolute average Number of threads used by vCenter
Received packet rate Number rate average Rate of the number of total packets received per second on the system where vCenter is running
Sent packet rate Number rate average Number of total packets sent per second on the system where vCenter is running
System call rate Number rate average Number of systems calls made per second on the system where vCenter is running
System threads Number absolute average Number of threads on the system where vCenter is running
Total packet rate Number rate average Number of total packets sent and received per second on the system where vCenter is running
Virtual memory KB absolute average Virtual memory used by vCenter

vSphere Replication

MetricUnitStats TypeRollupDescription
Replication Data Receive Rate KBps rate average Average amount of data received per second
Replication Data Transmit Rate KBps rate average Average amount of data transmitted per second
vSphere Replication VM Count Number absolute average Current Number of Replicated VMs