Monday, June 24, 2013

Riak CAP Tuning and F#

Riak provides the ability to tune CAP. CAP, which stands for Consistency, Availability, and Partition tolerance, does not seem like controls that are tunable. These terms seem evoke images of binary choices, as in either you have it or you don't. CAP terms by itself is ambiguous in their definitions. I'm not the only one who feels that way as can be seen in Daniel Abadi's blog post. For me, it was more helpful for me to think of tradeoffs as consistency latency (time needed to achieve eventual consistency), performance (read/write latency), and node failure tolerance (how many nodes can fail and still have a working cluster).

Riak exposes their CAP tuning controls via the named variables N, R, and W. These variables are defined as follows:

N
Number of nodes to replicated a piece of data
R
Number of nodes to read data to be considered success (read failure tolerance)
W
Number of nodes to write data to be considered write complete (write fault tolerance)

In addition, Riak exposes these additional tuning controls:

PR
Number of primary, non-fallback nodes that must return results for a successful read
PW
Number of primary, non-fallback nodes that must accept a write
DW
Number of nodes which have received an acknowledgement of the write from the storage backend

Bucket Level CAP Controls in Riak

Here's an example on how to set bucket level CAP settings in Riak with CorrugatedIron:


// Get existing bucket properties
let properties = ciClient.GetBucketProperties("animals",true).Value

// Set # of nodes a write must ultimately replicate to
// This should be set at the creation of the bucket
properties.SetNVal(3u)

// Set number of nodes that must successfully written before successful write response
properties.SetWVal(2u)

// Set # of nodes required to read a value succesfully
properties.SetRVal(1u)

// Set primary read value
properties.SetPrVal(1u)

// Set primary write value
properties.SetPwVal(1u)

// Set durable write value
properties.SetDwVal(1u)

// Change bucket properties with these new CAP control values
ciClient.SetBucketProperties("animals",properties)

Per Request CAP Controls in Riak

Riak allows you to tune CAP controls at per request level:

// Setting W & DW on puts
let options = new RiakPutOptions()
options.SetW(3u).SetDw(1u)
let data = new RiakObject("animals","toto",{nickname="Toto"; breed="Cairn Terrier"; score=5})
ciClient.Put(data,options)

// Get item with R value set to 2
ciClient.Get("animals","toto",1u).Value.GetObject<Animal>()

// Specify quorum
let getOptions = new RiakGetOptions()
getOptions.SetR("quorum")

// Need to convert IRiakClient to RiakClient in order to set RiakGetOptions
let client = ciClient :?> RiakClient
client.Get("animals","toto",getOptions).Value.GetObject<Animal>()

Monday, June 03, 2013

Riak MapReduce with F#

Lately, I have been reading the book Signals and Noise by Nate Silver. His book references an IBM webpage that claims the world is creating 2.5 quintillion (1018) bytes of data a day and that 90% of the data that currently exists the world was created in the past two years. Combine this fact with the rise in multicore processors, would explain the surging interest in functional programming and NoSQL and MapReduce frameworks. There is just too much data to be handled by a single machine alone

Continuing the journey in the book Seven Databases in Seven Weeks as I explore MapReduce. MapReduce framework is popularized by Google's seminal paper, but the genesis of that framework started with ideas in functional programming languages. After all, if functional programming is intended to help one run code on multiple processors on a single server, it's a logical conclusion to extend that computing from multiple processors on a single server to multiple servers. However, it's the recent technology factors and the data tsunami that I believe encourages people to look at functional programming and NoSQL databases.

What I would like to see in the future is that MapReduce capability is integrated with functional programming in such a way that calling a map/reduce function on a typed collection that recognizes that the typed collection is backed by a NoSQL database that it seamlessly spawns the function to the NoSQL stores in a MapReduce fashion.

I want to do something like the following:

Seq.map myfunc myRiakCollection

and the compiler/runtime will automatically convert myfunc into an appropriate form to run on all the data nodes and returns the results. But since we're not there yet, we need to do a little more work.

Some additional items to note, Riak Handbook warns that:

MapReduce in Riak is not exactly the same as Hadoop, where you can practically analyze an infinite amount of data. In Riak, data is pulled out of the system to be analyzed. That alone sets the boundaries of what's possible.

This is further reinforced by this following mailing list entry in a response from Sean Cribbs, developer advocate from Basho:

...Riak's MapReduce is designed for low-latency queries, but it's also designed for "targeted" queries where you don't slurp the entire contents of a bucket from disk. Structure your data so that you can know -- or deterministically guess -- what keys should be involved in your query. Then you won't be trying to fetch and filter all of the data all the time.

I wanted to point this out as this goes against my mental model of how MapReduce should work.


MapReduce with CorrugatedIron

In this particular example, I will use the Javascript example from the book. First, we'll define some helper functions that provides some syntatic sugar in working with the CorrugatedIron's fluent interface:

// Helper function for CorrugatedIron's job inputs for bucket/key pairs only.
let setinputs data (q:RiakMapReduceQuery) = 
    let inputs = new RiakBucketKeyInput()

    data
    |> List.iter( fun (bucket,key) -> inputs.AddBucketKey(bucket,key)) 

    q.Inputs(inputs)

//  Helper function mapping Javascript functions
let mapjs  src keep (q:RiakMapReduceQuery) =
    q.MapJs(fun m -> m.Source(src).Keep(keep) |> ignore)

// Helper function for mapping stored functions
let mapsf bucket func keep (q:RiakMapReduceQuery) =
    q.MapJs(fun m -> m.BucketKey(bucket,func).Keep(keep) |> ignore)

// Helper function for mapping builtin functions
let mapbuiltins func keep (q:RiakMapReduceQuery) =
    q.MapJs(fun m -> m.Name(func).Keep(true) |> ignore)

// Getting results from CorrugatedIron seems convoluted to me
// Helper function is used to extract the data to a simpler form 
let getResults (results:RiakResult<RiakMapReduceResult>) =
    let getText raw = raw |> System.Text.Encoding.Default.GetString
    results.Value.PhaseResults 
 |> Seq.map (fun phase -> phase.Values |> Seq.map getText ) 
 |> Seq.concat 
 
// Helper function for Reduce operations in Javascript
let reducejs src keep (q:RiakMapReduceQuery) =
    q.ReduceJs(fun m -> m.Source(src).Keep(keep) |> ignore)

// Helper function for link walking with mapreduce
let linkphase bucket keep (q:RiakMapReduceQuery) =
    q.Link(fun l -> l.Keep(keep).Bucket(bucket) |> ignore)
 

Here's the script to perform the MapReduce in CorrugatedIrons. In the following snippet, ciClient refers to CorrugatedIron's RiakClient object, in the previous blogs, ciClient was just client:

// Mapreduce example from the book
let src ="function(v) {
     /* From the Riak object, pull data and parse it as JSON */
     var parsed_data = JSON.parse(v.values[0].data);
     var data = {};

     /* Key capacity number by room style string */
     data[parsed_data.style] = parsed_data.capacity;
     return [data];
   }"

let rooms =  [("rooms","101");("rooms","102");("rooms","103");]

let query = new RiakMapReduceQuery()
   |> setinputs rooms
   |> mapjs src true

ciClient.MapReduce(query)
|> getResults

Results:

seq ["[{"suite":3}]"; "[{"king":1}]"; "[{"double":4}]"]

MapReduce with REST API

It's not all that hard to do MapReduce with ASP.NET MVC REST API although your json script becomes more complex. In the following example, I had to escape the double quote(") because, alas, I'm still using F# 2.0. If you're using F# 3.0, you can use the new triple quote string syntax to wrap the MapReduce script.

// This helper function will be used in all the later examples with REST API
let mapred script =
    let post_url = sprintf "%s/mapred" riakurl 
    let content = new StringContent(script)
    content.Headers.ContentType.MediaType <- "application/json"
    let response = restClient.PostAsync(post_url,content)
    response.Wait()
    let results = response.Result.Content.ReadAsStringAsync()
    results

let mapreduceScript =  "{
    \"inputs\":[
        [\"rooms\",\"101\"],[\"rooms\",\"102\"],[\"rooms\",\"103\"]
    ],
    \"query\":[
        {\"map\":{
            \"language\":\"javascript\",
            \"source\":
            \"function(v) {
                /* From the Riak object, pull data and parse it as JSON */
                var parsed_data = JSON.parse(v.values[0].data);
                var data = {};
                /* Key capacity number by room style string */
                data[parsed_data.style] = parsed_data.capacity;
                return [data];
            }\"
        }}
      ]
    }"

mapred mapreduceScript 

Results:

val it : Task<string> =
  System.Threading.Tasks.Task`1[System.String]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 2;
     IsCanceled = false;
     IsCompleted = true;
     IsFaulted = false;
     Result = "[{"king":1},{"suite":3},{"double":4}]";
     Status = RanToCompletion;}


Stored Functions with CorrugatedIron

I can store custom Javascript mapreduce functions in a specific bucket/key and call it for use:

let myfunc = "
 function(v) {
  var parsed_data = JSON.parse(v.values[0].data);
  var data = {};
  data[parsed_data.style] = parsed_data.capacity;
  return [data];
 }"

// Store the map function in a bucket value
let putResults = new RiakObject("my_functions","map_capacity", myfunc) 
                 |> ciClient.Put

// Querying using stored map function
let rooms = [("rooms","101");("rooms","102");("rooms","103");("rooms","104");]

let query = new RiakMapReduceQuery()
   |> setinputs rooms
   |> mapsf "my_functions" "map_capacity" true

ciClient.MapReduce(query)
|> getResults

Results:

seq ["[{"suite":3}]"; "[{"king":1}]"; "[{"double":4}]"; "[{"suite":7}]"]

Stored Functions with REST API

With REST API, reusing the mapred function defined previously:

let storedFuncScript = "
{
    \"inputs\":[
        [\"rooms\",\"101\"],[\"rooms\",\"102\"],[\"rooms\",\"103\"],[\"rooms\",\"104\"]
    ],
    \"query\":[
        {\"map\":{
        \"language\":\"javascript\",
        \"bucket\":\"my_functions\",
        \"key\":\"map_capacity\"
    }}
    ]
}
"

let savesf bucket key myfunc =
    let post_url = sprintf "%s/riak/%s/%s" riakurl bucket key
    let content = new StringContent(myfunc)
    content.Headers.ContentType.MediaType <- "application/json"
    restClient.PostAsync(post_url,content)

// Store the map function in a bucket value
savesf "my_functions" "map_capacity" myfunc


mapred storedFuncScript

Results:

val it : TaskTask<string> =
  System.Threading.Tasks.Task`1[System.String]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 5;
     IsCanceled = false;
     IsCompleted = true;
     IsFaulted = false;
     Result = "[{"suite":3},{"double":4},{"king":1},{"suite":7}]";
     Status = RanToCompletion;}

Built-in Functions with CorrugatedIron

Riak has some built-in mapreduce javascript functions. I couldn't find any reference documentation on it, the closes I came is the actual Javascript code in github: . Here are the list of builtin Javascript functions from that source code:

  • Riak.getClassName(obj)
  • Riak.filterNotFound(values)
  • Riak.mapValues(value,keyData,arg)
  • Riak.mapValuesJson(value,keyData,arg)
  • Riak.mapByFields(value,keyData,fields)
  • Riak.reduceSum(values,arg)
  • Riak.reduceMin(values,arg)
  • Riak.reduceMax(values,arg)
  • Riak.reduceSort(value,arg)
  • Riak.reduceNumericSort(value, arg)
  • Riak.reduceLimit(value, arg)
  • Riak.reduceSlice(value, arg)

In the following example, we're only going to use Riak.mapValuesJson():

let rooms = [("rooms","101");("rooms","102");("rooms","103");("rooms","104");]


let query = new RiakMapReduceQuery()
   |> setinputs rooms
   |> mapbuiltins "Riak.mapValuesJson" true

let results = ciClient.MapReduce(query)

// Collect the results and flatten the nested collections
results.Value.PhaseResults
|> Seq.map (fun x -> x.GetObjects<IEnumerable<Room>>())
|> Seq.concat
|> Seq.concat

Results:

val it : seq<Room> =
  seq [{style = "double"; capacity = 4;}; 
       {style = "suite";  capacity = 3;}; 
    {style = "suite";  capacity = 7;}; 
    {style = "king";   capacity = 1;}]

Built-in Functions with CorrugatedIron

Calling built-in functions with REST API:

let builtinFunc = "
{
 \"inputs\":[
  [\"rooms\",\"101\"],[\"rooms\",\"102\"],[\"rooms\",\"103\"]
 ],
 \"query\":[
  {\"map\":{
  \"language\":\"javascript\",
  \"name\":\"Riak.mapValuesJson\"
 }}
 ]
}
"

mapred builtinFunc

Results:

val it : Task<string> =
  System.Threading.Tasks.Task`1[System.String]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 6;
     IsCanceled = false;
     IsCompleted = true;
     IsFaulted = false;
     Result = "[{"style":"double","capacity":4},
             {"style":"king","capacity":1},
    {"style":"suite","capacity":3}]";
     Status = RanToCompletion;}

Reducing with CorrugatedIron

Reduce operations with CorrguatedIron:

let reduceScript = "
function(v) {
  var totals = {};
  for (var i in v) {
 for(var style in v[i]) {
   if (totals[style]) totals[style] += v[i][style];
   else               totals[style] = v[i][style];
   }
  }
  return [totals];        
}"


let query = (new RiakMapReduceQuery()).Inputs("rooms")
   |> mapsf "my_functions" "map_capacity" false
   |> reducejs reduceScript true


ciClient.MapReduce(query)
|> getResults

Results:

seq["[{"queen":9051,"king":9189,"single":8811,"double":8629,"suite":9231}]"]

Reduce operations with REST API

MapReduce with REST API:

let myReduceScript = "
{
    \"inputs\":\"rooms\",
    \"query\":[
    {\"map\":{
        \"language\":\"javascript\",
        \"bucket\":\"my_functions\",
        \"key\":\"map_capacity\"
    }},
    {\"reduce\":{
        \"language\":\"javascript\",
        \"source\":
            \"function(v) {
                var totals = {};
                for (var i in v) {
                    for(var style in v[i]) {
                        if( totals[style] ) totals[style] += v[i][style];
                        else totals[style] = v[i][style];
                    }
                }
                return [totals];
            }\"
        }}
    ]
}"

mapred myReduceScript

Results:

val it : Task<string> =
  System.Threading.Tasks.Task`1[System.String]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 7;
     IsCanceled = false;
     IsCompleted = true;
     IsFaulted = false;
     Result = "[{"suite":9231,"king":9189,"double":8629,"queen":9051,"single":8811}]";
     Status = RanToCompletion;}

Key Filters with CorrugatedIron

Key filters is used to reduce the input set before the map/reduce operations:

let reduceScript = "
 function(v) {
   var totals = {};
   for (var i in v) {
  for(var style in v[i]) {
    if (totals[style]) totals[style] += v[i][style];
    else               totals[style] = v[i][style];
    }
   }
   return [totals];        
}"

let keyFilter = new Action<RiakFluentKeyFilter>(fun x -> x.StringToInt().LessThan(1000) |> ignore)

let query = (new RiakMapReduceQuery()).Inputs("rooms").Filter(keyFilter)
   |> mapsf "my_functions" "map_capacity" false
   |> reducejs reduceScript true


ciClient.MapReduce(query)
|> getResults

Results:

val it : seq<string> =
  seq ["[{"queen":780,"suite":968,"king":838,"single":791,"double":758}]"]

Key Filters with REST API

Adding key filters with REST API:

let keyFilterScript = "
{
    \"inputs\":{
        \"bucket\":\"rooms\",
        \"key_filters\":[[\"string_to_int\"], [\"less_than\", 1000]]},    
    \"query\":[
    {\"map\":{
        \"language\":\"javascript\",
        \"bucket\":\"my_functions\",
        \"key\":\"map_capacity\"
    }},
    {\"reduce\":{
        \"language\":\"javascript\",
        \"source\":
            \"function(v) {
                var totals = {};
                for (var i in v) {
                    for(var style in v[i]) {
                        if( totals[style] ) totals[style] += v[i][style];
                        else totals[style] = v[i][style];
                    }
                }
                return [totals];
            }\"
        }}
    ]
}"

mapred keyFilterScript

Results:

val it : Task<string> =
  System.Threading.Tasks.Task`1[System.String]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 10;
     IsCanceled = false;
     IsCompleted = true;
     IsFaulted = false;
     Result = "[{"queen":780,"king":838,"suite":968,"single":791,"double":758}]";
     Status = RanToCompletion;}

MapReduce Link Walking with CorrugatedIron

In my previous blog, I talked about the fact I couldn't figure out how to specify the keep flag in the link walk examples. With map reduce link walking in CorrugatedIron, I now can now specify the keep flag for each of the link-map-reduce phases:


let keyFilter = new Action<RiakFluentKeyFilter>(fun x -> x.Equal("2") |> ignore)

let query = (new RiakMapReduceQuery()).Inputs("cages").Filter(keyFilter)
   |> linkphase "animals" false
   |> mapjs "function(v) { return [v]; }" true

ciClient.MapReduce(query)
|> getResults

Results:


val it : seq<seq<string>> =
  seq
    [seq [];
     seq ["[{
    "bucket":"animals",
    "key":"ace",
    "vclock":"a85hYGBgymDKBVIcrFrhrwI5O5wzmBKZ8lgZlj/hPM0HlVLm/PczkDP1FlRqHUgqCwA=",
    "values":
      [{"metadata":
     {"X-Riak-VTag":"4XZxrmgvKzan8X4FKz5hti",
      "content-type":"application/json",
   "index":[],
   "X-Riak-Last-Modified":"Thu, 16 May 2013 23:15:58 GMT"},
   "data":"{\"nickname\":\"The Wonder Dog\",\"breed\":\"German Shepherd\"}"}]}]"]] 

MapReduce Link Walking with REST API

With REST API:

// Code for Link Walking with MapReduce
let mrLinkWalk = "
{
 \"inputs\":{
  \"bucket\":\"cages\",
  \"key_filters\":[[\"eq\", \"2\"]]
 },
 \"query\":[
  {\"link\":{
   \"bucket\":\"animals\",
   \"keep\":false
  }},
  {\"map\":{
   \"language\":\"javascript\",
   \"source\":
    \"function(v) { return [v]; }\"
  }}
 ]
}
"

mapred mrLinkWalk

Results:

val it : Task<string> =
  System.Threading.Tasks.Task`1[System.String]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 13;
     IsCanceled = false;
     IsCompleted = true;
     IsFaulted = false;
     Result = "[{
   "bucket":"animals",
   "key":"ace",
   "vclock":"a85hYGBgymDKBVIcrFrhrwI5O5wzmBKZ8lgZlj/hPM0HlVLm/PczkDP1FlRqHUgqCwA=",
   "values":[{
     "metadata":{
    "X-Riak-VTag":"4XZxrmgvKzan8X4FKz5hti",
    "content-type": "application/json",
    "index":[],
    "X-Riak-Last-Modified":"Thu, 16 May 2013 23:15:58 GMT"},
      "data":"{\"nickname\":\"The Wonder Dog\",\"breed\":\"German Shepherd\"}"}]}]";
     Status = RanToCompletion;}

Sunday, May 12, 2013

Riak Links and Link Walking with F# and CorrugatedIron

Continuing my journey through the book Seven Databases in Seven Weeks, I explore links and link walking in this blog post. Riak has the ability to establish one-way relationship between entries via Links, providing some of the capabilities of a graph database (Riak documentation calls it a lightweight graph database). Riak documentation hints that links should be kept low, on the order of dozens, not thousands. Using the Twitter example from Riak Handbook, you can probably easily find out who Don Syme follows and who does those people follows, etc. But it would be difficult to find all the people that follows Don Syme using Riak's link capability.


Adding Links

Here's how you would add links with CorrugatedIron library:

type Cage = { room : int }

(*
Linking cage 1 to polly via contains, equivalent to doing the following
curl -X PUT http://localhost:8098/riak/cages/1 \
-H "Content-Type: application/json" \
-H "Link: </riak/animals/polly>; riaktag=\"contains\"" \
-d '{"room" : 101}'
*)
client.Put(
  let cage = new RiakObject("cages","1",{room=101})
  cage.LinkTo("animals","polly","contains")
  cage)

(*
Putting ace in cage 2 and setting cage 2 next to cage 1.
Equivalent to the following sample code:
curl -X PUT http://localhost:8091/riak/cages/2 \
-H "Content-Type: application/json" \
-H "Link:</riak/animals/ace>;riaktag=\"contains\",</riak/cages/1>;riaktag=\"next_to\"" \
-d '{"room" : 101}'
*)
// Adding more than one link  
client.Put(
  let cage = new RiakObject("cages","2",{room=101})
  [("animals","ace","contains");("cages","1","next_to")]
  |> List.iter(fun (bucket,key,tag) -> cage.LinkTo(bucket,key,tag))
  cage)  


Link Walking

With CorrugatedIron, I can use the API to perform link walking and as an extra bonus, I don't have to worry about extracting data from the multipart/mixed mime types. CorrugateIron library takes care of all of that for us.

(*
Link walking, equivalent to 
curl http://riakhost1:8098/riak/cages/1/_,_,_
or in the new version:
curl http://riakhost1:8098/buckets/cages/keys/1/_,_,_
*)
 
let results = client.WalkLinks(new RiakObject("cages","1"),
                               [|new RiakLink(null,null,null)|])
         
// Dump results, which returns a list of RiakObject(s)
results.Value

// Since in this particular case, we have only one result, we can do the following
// and get back polly
results.Value.[0].GetObject<Animal>()

Here is the output result for results.Value:

// results.Value
val it : IList<RiakObject> =
  seq
    [CorrugatedIron.Models.RiakObject
       {BinIndexes = dict [];
        Bucket = "animals";
        CharSet = null;
        ContentEncoding = null;
        ContentType = "application/json";
        HasChanged = false;
        IntIndexes = dict [];
        Key = "polly";
        LastModified = 1359758822u;
        LastModifiedUsec = 523420u;
        Links = seq [];
        Siblings = seq [];
        UserMetaData = dict [];
        VTag = "2BTveSKTYDNOZNCiOxyryw";
        VTags = seq ["2BTveSKTYDNOZNCiOxyryw"];
        Value = [|123uy; 32uy; 34uy; 110uy; 105uy; 99uy; 107uy; 110uy; 97uy;
                  109uy; 101uy; 34uy; 32uy; 58uy; 32uy; 34uy; 83uy; 119uy;
                  101uy; 101uy; 116uy; 32uy; 80uy; 111uy; 108uy; 108uy; 121uy;
                  32uy; 80uy; 117uy; 114uy; 101uy; 98uy; 114uy; 101uy; 100uy;
                  34uy; 32uy; 44uy; 32uy; 34uy; 98uy; 114uy; 101uy; 101uy;
                  100uy; 34uy; 32uy; 58uy; 32uy; 34uy; 80uy; 117uy; 114uy;
                  101uy; 98uy; 114uy; 101uy; 100uy; 34uy; 32uy; 125uy|];
        VectorClock = [|107uy; 206uy; 97uy; 96uy; 96uy; 96uy; 204uy; 96uy;
                        202uy; 5uy; 82uy; 28uy; 169uy; 111uy; 239uy; 241uy;
                        7uy; 114uy; 230uy; 184uy; 103uy; 48uy; 37uy; 50uy;
                        230uy; 177uy; 50uy; 60uy; 59uy; 216uy; 112uy; 138uy;
                        47uy; 11uy; 0uy|];}]
      
// results.Value.[0].GetObject<Animal>()
val it : Animal = {nickname = "Sweet Polly Purebred";
                   breed = "Purebred";}     

There seems to be some limitations with link walking using CorrugatedIron library. There is a WalkLinks() method as part RiakClient, but the input is expecting RiakLink object which has bucket,key, tag as arguments. The link spec is expecting bucket, tag, and keep flag as arguments. I do notice that in the Riak documentation that Link Walking is not available as part of the Protocol Buffers Client (PBC) API, so I'm guessing the WalkLinks() method in CorrugatedIron is either using HTTP protocol or a modified usage of MapReduce. Since link walking is a special case of MapReduce querying, it may not matter much that CorrugatedIron has some limitations on link walking. One other issue with CorrugatedIron and link walking is that when I add multiple links with the same tag and try to link walking with CorrugatedIron, I do not get back all the links, I only get one of the links. In order to follow the examples in the book Seven Databases in Seven Weeks, I can fall back to using ASP.NET MVC Rest API.

Link walking using CorrugatedIron library:

// Link walking, equivalent to 
// curl http://localhost:8098/riak/cages/2/animals,_,_
client.WalkLinks(new RiakObject("cages","2"),
                 [|new RiakLink("animals",null,null)|])
     
// curl http://localhost:8098/riak/cages/2/_,next_to,0/animals,_,_
client.WalkLinks(new RiakObject("cages","2"),
                 [|new RiakLink(null,null,"next_to");
                   new RiakLink("animals",null,null);|])

// I can't seem to specify the keep flag with CorrugatedIron, which keeps
// intermediate results as you walk beyond primary links
// curl http://localhost:8091/riak/cages/2/_,next_to,1/_,_,_     

The book Seven Databases in Seven Weeks is still using the old format for HTTP Link Walking. The new format is as follows:

GET /riak/bucket/key/[bucket],[tag],[keep]            # Old format
GET /buckets/bucket/keys/key/[bucket],[tag],[keep]    # New format

Link walking using REST API:


let riakurl = "http://myriakhost1:8098"
let restClient = new HttpClient()

type LinkWalkSpec = 
    { bucket: string; tag: string;  keep: string; }

    member x.Link = (sprintf "%s,%s,%s" x.bucket x.tag x.keep)

let linkWalker url bucket key (links:LinkWalkSpec list) =
    let baseurl = sprintf "%s/buckets/%s/keys/%s" url bucket key
    let rec buildLinkWalkUrl (linklist:LinkWalkSpec list) baseUrl =
      match linklist with
      | [] -> baseUrl
      | [x] -> sprintf "%s/%s" baseUrl (x.Link)
      | h::t -> let newUrl = sprintf "%s/%s" baseUrl (h.Link)
                buildLinkWalkUrl t newUrl

    buildLinkWalkUrl links baseurl
    |> restClient.GetStringAsync

// Equiv to : curl http://localhost:8091/riak/cages/2/_,next_to,1/_,_,_  
   
[{bucket="_";tag="next_to";keep="1"};{bucket="_";tag="_";keep="_"}] 
|> linkWalker riakurl "cages" "2" 

Link walking results from using REST API:

val it : Task<string> =
  System.Threading.Tasks.Task`1[System.String]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 1;
     IsCanceled = false;
     IsCompleted = false;
     IsFaulted = false;
     Result = "
--CveXyss6PAqBxOOWxeWBCf6eXii
Content-Type: multipart/mixed; boundary=AvYXKrJYDlkeNxh1bQyqDvBAuBF

--AvYXKrJYDlkeNxh1bQyqDvBAuBF
X-Riak-Vclock: a85hYGBgzGDKBVIcqW/v8Qdy5rhnMCUy5rEy9Oi+P8WXBQA=
Location: /buckets/cages/keys/1
Content-Type: application/json
Link: </buckets/animals/keys/polly>; riaktag="contains", </buckets/cages>; rel="up"
Etag: 6gyXgkIgzvwBGRRotqHK3b
Last-Modified: Fri, 26 Apr 2013 16:55:40 GMT

{"room":101}
--AvYXKrJYDlkeNxh1bQyqDvBAuBF--

--CveXyss6PAqBxOOWxeWBCf6eXii
Content-Type: multipart/mixed; boundary=SaWqmmho48dzhMDmJy3BVcCWrzu

--SaWqmmho48dzhMDmJy3BVcCWrzu
X-Riak-Vclock: a85hYGBgzGDKBVIcqW/v8Qdy5rhnMCUy5rEyPDvYcIovCwA=
Location: /buckets/animals/keys/polly
Content-Type: application/json; charset=utf-8
Link: </buckets/animals>; rel="up"
Etag: 2BTveSKTYDNOZNCiOxyryw
Last-Modified: Fri, 01 Feb 2013 22:47:02 GMT

{ "nickname" : "Sweet Polly Purebred" , "breed" : "Purebred" }
--SaWqmmho48dzhMDmJy3BVcCWrzu--

--CveXyss6PAqBxOOWxeWBCf6eXii--
";
     Status = RanToCompletion;}      

Friday, April 26, 2013

Exploring Riak with F# and CorrugatedIron

Many thanks to David and OJ for recommending CorrugatedIron .Net Riak client library. The following blog entry is to document my experiments with Riak CRUD operations using CorrugatedIron.

Pinging RIAK

I tested CorrugatedIron with F# script and here is the setup code along with testing ping capability:

// Needed to load the following libraries to get F# script to work
#r @"c:\dev\FsRiak\packages\CorrugatedIron.1.3.0\lib\net40\CorrugatedIron.dll"
#r @"c:\dev\FsRiak\packages\protobuf-net.2.0.0.621\lib\net40\protobuf-net.dll"
#r @"c:\dev\FsRiak\packages\Newtonsoft.Json.4.5.11\lib\net40\Newtonsoft.Json.dll"

open CorrugatedIron
open CorrugatedIron.Models
open Newtonsoft.Json
open System

// Setup connections
let cluster = RiakCluster.FromConfig("riakConfig", @"c:\dev\FsRiak\App.config");
let client = cluster.CreateClient();

// Ping the Riak Cluster
client.Ping()

Here is my App.config file used by CorrugatedIron:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="riakConfig" type="CorrugatedIron.Config.RiakClusterConfiguration, CorrugatedIron" />
  </configSections>
  <riakConfig nodePollTime="5000" defaultRetryWaitTime="200" defaultRetryCount="3">
    <nodes>
      <node name="dev1"  hostAddress="mydevhost-a" pbcPort="8087" restScheme="http" restPort="8098" poolSize="10" />
      <node name="dev2" hostAddress="mydevhost-b" pbcPort="8087" restScheme="http" restPort="8098" poolSize="10" />
      <node name="dev3" hostAddress="mydevhost-c" pbcPort="8087" restScheme="http" restPort="8098" poolSize="10" />
    </nodes>
  </riakConfig>
</configuration>

Here's the result from running ping:

val it : RiakResult = CorrugatedIron.RiakResult {ErrorMessage = null;
                                                 IsSuccess = true;
                                                 ResultCode = Success;}

Get List of Buckets

The following method call gets you the list of buckets along with metadata for the call status:

client.ListBuckets()

This method returns the following RiakResult object:

val it : RiakResult<seq<string>> =
  CorrugatedIron.RiakResult`1[System.Collections.Generic.IEnumerable`1[System.String]]
    {ErrorMessage = null;
     IsSuccess = true;
     ResultCode = Success;
     Value = seq ["photos"; "favs"; "animals"; "cages"; ...];}

Get Bucket Keys

Getting a list of keys for a bucket is also simple:

client.ListKeys("animals")

For the novice, this library warns you to not to do this in production environments....

*** [CI] -> ListKeys is an expensive operation and should not be used in Production scenarios. ***
val it : RiakResult<seq<string>> =
  CorrugatedIron.RiakResult`1[System.Collections.Generic.IEnumerable`1[System.String]]
    {ErrorMessage = null;
     IsSuccess = true;
     ResultCode = Success;
     Value = seq ["ace"; "polly"];}

Retrieve Content from Riak

Getting a value from Riak is pretty easy with this library:

client.Get("animals","ace")

A dump of the return object shows the actual data plus metadata about the Get operation:

val it : RiakResult<Models.RiakObject> =
  CorrugatedIron.RiakResult`1[CorrugatedIron.Models.RiakObject]
    {ErrorMessage = null;
     IsSuccess = true;
     ResultCode = Success;
     Value = CorrugatedIron.Models.RiakObject;}

A deeper dive into the Value field of RiakResult object gives the following:

val it : Models.RiakObject =
  CorrugatedIron.Models.RiakObject
    {BinIndexes = dict [];
     Bucket = "animals";
     CharSet = null;
     ContentEncoding = null;
     ContentType = "application/json";
     HasChanged = false;
     IntIndexes = dict [];
     Key = "ace";
     LastModified = 1359744019u;
     LastModifiedUsec = 788127u;
     Links = seq [];
     Siblings = seq [];
     UserMetaData = dict [];
     VTag = "7aPFusRQHlQ36ZP6G6GSyE";
     VTags = seq ["7aPFusRQHlQ36ZP6G6GSyE"];
     Value = [|123uy; 32uy; 34uy; 110uy; 105uy; 99uy; 107uy; 110uy; 97uy;
               109uy; 101uy; 34uy; 32uy; 58uy; 32uy; 34uy; 84uy; 104uy; 101uy;
               32uy; 87uy; 111uy; 110uy; 100uy; 101uy; 114uy; 32uy; 68uy;
               111uy; 103uy; 34uy; 32uy; 44uy; 32uy; 34uy; 98uy; 114uy; 101uy;
               101uy; 100uy; 34uy; 32uy; 58uy; 32uy; 34uy; 71uy; 101uy; 114uy;
               109uy; 97uy; 110uy; 32uy; 83uy; 104uy; 101uy; 112uy; 104uy;
               101uy; 114uy; 100uy; 34uy; 32uy; 125uy|];
     VectorClock = [|107uy; 206uy; 97uy; 96uy; 96uy; 96uy; 204uy; 96uy; 202uy;
                     5uy; 82uy; 28uy; 172uy; 90uy; 225uy; 175uy; 2uy; 57uy;
                     59uy; 156uy; 51uy; 152uy; 18uy; 25uy; 243uy; 88uy; 25uy;
                     132uy; 59uy; 26uy; 78uy; 241uy; 101uy; 1uy; 0uy|];}

The data I really wanted is embedded in another Value field where it is represented as an array of bytes, which is not the final format I want. I wanted to get back the JSON representation of the data I put in. In my previous blog, I had rolled my own JSON serializer/deserializer, but now I wanted to leverage the other pieces of library bundle by CorrugateIron, namely Json.NET. To do so,I define the Animal type and use Json.NET serializer/deserializer to convert between the objects and it's corresponding JSON representation

type Animal =
    { nickname: string; breed: string}

// Getting ace from animals bucket 
client.Get("animals","ace").Value.GetObject<Animal>() 

Adding Content to Riak

Adding content to Riak is pretty easy also, after you define a specific type for Json.NET to serialize the fields of that type:

new RiakObject("animals","delta",{nickname="Snoopy"; breed="Beagle"}) 
|> client.Put

If you dump the RiakResult object and the Value field of RiakResult you get the following:


val it : RiakResult =
  CorrugatedIron.RiakResult`1[CorrugatedIron.Models.RiakObject]
    {ErrorMessage = null;
     IsSuccess = true;
     ResultCode = Success;
     Value = CorrugatedIron.Models.RiakObject;}
  
val it : RiakObject =
  CorrugatedIron.Models.RiakObject
    {BinIndexes = dict [];
     Bucket = "animals";
     CharSet = null;
     ContentEncoding = null;
     ContentType = "application/json";
     HasChanged = false;
     IntIndexes = dict [];
     Key = "delta";
     LastModified = 1366930771u;
     LastModifiedUsec = 271921u;
     Links = seq [];
     Siblings = seq [];
     UserMetaData = dict [];
     VTag = "OvZlH7bsYKdO8zL76QdDY";
     VTags = seq ["OvZlH7bsYKdO8zL76QdDY"];
     Value = [|123uy; 34uy; 110uy; 105uy; 99uy; 107uy; 110uy; 97uy; 109uy;
               101uy; 34uy; 58uy; 34uy; 83uy; 110uy; 111uy; 111uy; 112uy;
               121uy; 34uy; 44uy; 34uy; 98uy; 114uy; 101uy; 101uy; 100uy; 34uy;
               58uy; 34uy; 66uy; 101uy; 97uy; 103uy; 108uy; 101uy; 34uy; 125uy|];
     VectorClock = [|107uy; 206uy; 97uy; 96uy; 96uy; 96uy; 204uy; 96uy; 202uy;
                     5uy; 82uy; 28uy; 169uy; 111uy; 239uy; 241uy; 7uy; 114uy;
                     230uy; 184uy; 103uy; 48uy; 37uy; 50uy; 230uy; 177uy; 50uy;
                     4uy; 27uy; 190uy; 59uy; 197uy; 151uy; 5uy; 0uy|];}

To verify interoperability, I would check the newly added data with curl:

$ curl -X GET http://192.168.56.1:8098/riak/animals/delta
$ {"nickname":"Snoopy","breed":"Beagle"}

Delete Riak Contents

Delete content by calling the intuitively named Delete method:

client.Delete("animals","delta")

One thing that I wasn't sure is how CorrugatedIron talks to the Riak clusters. In my simple REST API example, I know exactly which host I'm talking to since I explicitly specified the url. For CorrugatedIron, I configure a pool of connections in the app.config file. I wasn't sure which of the nodes CorrugatedIron was talking to. I fire up Wireshark and notice that I'm connected to the Riak cluster via port 8087, which is through the Protocol Buffers Client (PBC) interface...which explains the need to load the PBC libraries. This Protocol Buffers client is something new to me and the bundled protobuf-net library seemed to be the code developed by Google (Many thanks to OJ for correcting me on this, this is NOT a Google library but a library written by Marc Gravell). This, in turn led me to look at Google Protocol Buffers . This little side jaunt into CorrugatedIron library led to the discovery (for me) of a whole new set of network communication protocols. In any case, after checking Wireshark output, it seems that the requests are spread out to the different nodes and not restricted to a single node. I'm guessing that there are some builtin load balancer code in CorrugatedIron that sends my request to different nodes in the Riak cluster.

For the basic CRUD operations on Riak, CorrugatedIron has made it easier to work with Riak then having to come up with my own helper functions. This has been a good start and I hope work through more of the Riak examples from the book Seven Databases in Seven Weeks with CorrugatedIron in future blog posts.

Monday, April 15, 2013

Exploring Riak with F#

I have embraced the Polyglot Programming for quite a while already. This year, I wanted to tackle Polyglot Persistence. RDBMS has dominated my world view of persistence layer with everything else as second class citizens. I thought it was time to expand my world view of persistence layers, especially with the burgeoning popularity of NoSQL movement. To begin my exploration, I picked up the book Seven Databases in Seven Weeks by Eric Redmond and Jim Wilson and started with the first NoSQL persistence layer in the book, which was Riak. According to the book:

Riak is a distributed key-value database where values can be anything-from plain text, JSON, or XML to images or video clips-all accessible through a simple HTTP interface.

Setting up Riak

I deployed riak on 3 servers for testing purposes. In setting up the Riak clusters, I ran into the following errors:

10:42:57.339 [error] gen_server riak_core_capability terminated with reason: no function clause matching orddict:fetch('riak@192.168.56.1', [
{'riak@127.0.0.1',[{{riak_core,staged_joins},[true,false]},{{riak_core,vnode_routing},[proxy,legacy]},...]}]) line 72
/users/domains/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.

A quick Google search brought up the following link Googling the web, I got this link: http://blog.alwayshere.info/2012/11/riak-error-genserver-riakcorecapability.html I originally started riak with 127.0.0.1 address. Then I made the modification to the configuration as documented in http://docs.basho.com/riak/latest/cookbooks/Basic-Cluster-Setup/ in trying to setup my 3 server Riak into a cluster. To fix my error, I had to go to ./data/ring folder and delete everything in there, then everything works as expected.

Most of the examples in the book leveraged curl. However, I learn best if I tried to work the examples in another way. I tried the examples in Clojure, using ClojureWerkz's Welle. I liked ClojureWerkz's Welle wrappers to Riak and would probably use it if I had to develop on Java platform. I also wanted to work with Riak from .NET platform and hence I'm using F# to explore Riak. The following examples where done on Visual Studio 2010 with ASP.NET MVC 4 installed. This also gives me a chance to take the REST API in ASP.NET MVC4 for a spin.

Pinging RIAK

The very first example in the book is to ping the Riak cluster, here's how I implemented it in F#

open System.Net.Http
open System.Threading.Tasks

// My 3 instances of Riak
let riak1_url = "http://192.168.56.1:8098"
let riak2_url = "http://192.168.56.2:8098"
let riak3_url = "http://192.168.56.3:8098"

// Pick one to work with
let riakurl = riak1_url

let client = new HttpClient()

let ping () = client.GetAsync(sprintf "%s/ping" riakurl)

ping()

Running the ping, I would get the following response from F#:

val it : Task =
  System.Threading.Tasks.Task`1[System.Net.Http.HttpResponseMessage]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 4;
     IsCanceled = false;
     IsCompleted = false;
     IsFaulted = false;
     Result = StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Vary: Accept-Encoding
  Date: Wed, 30 Jan 2013 22:20:11 GMT
  Server: MochiWeb/1.1
  Server: WebMachine/1.9.0
  Server: (someone had painted it blue)
  Content-Length: 422
  Content-Type: application/json
};
     Status = RanToCompletion;}

Adding Content to RIAK

Let's start by putting some stuff into Riak with the following snippet of code

let put bucket key content =
    let put_url= sprintf "%s/riak/%s/%s" riakurl bucket key
    client.PutAsync(put_url,content)


let put_html bucket key html =
    let put_url= sprintf "%s/riak/%s/%s" riakurl bucket key
    let content = new StringContent(html)
    content.Headers.ContentType.MediaType <- "text/html"
    put bucket key content


"<html><body><h1>My latest favorite DB is RIAK</h1></body></html>"
|> put_html "favs" "db"

Running the above script gets the following response:

val it : Task =
  System.Threading.Tasks.Task`1[System.Net.Http.HttpResponseMessage]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 5;
     IsCanceled = false;
     IsCompleted = false;
     IsFaulted = false;
     Result = StatusCode: 204, ReasonPhrase: 'No Content', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Vary: Accept-Encoding
  Date: Fri, 01 Feb 2013 18:07:24 GMT
  Server: MochiWeb/1.1
  Server: WebMachine/1.9.0
  Server: (someone had painted it blue)
  Content-Length: 0
  Content-Type: text/html; charset=utf-8
};
     Status = RanToCompletion;}

We got the 204 code as explained in the book. To test that Riak has stored this new content, simply point to any of the Riak instances, (e.g. http://192.168.56.3:8098/riak/favs/db) with a browser and you should see the webpage that was put into the first Riak server.

Here's the sample code to put JSON data into Riak:

// Simple utility to generate JSON - should really use a real JSON library
let tojson data = 
    data |> Seq.map (fun (k,v) -> sprintf "\"%s\" : \"%s\"" k v)
         |> Seq.reduce (sprintf "%s , %s")
         |> sprintf "{ %s }" 


let put_json bucket key jsondata =
    let content = new StringContent(tojson jsondata)
    content.Headers.ContentType.MediaType <- "application/json"
    put bucket key content

[("nickname","The Wonder Dog"); ("breed","German Shepherd")]
|> put_json "animals" "ace"

Again, you can check that it's stored in Riak by pointing the browser to: http://192.168.56.3:8098/riak/animals/ace and you should get back:

{ "nickname" : "The Wonder Dog" , "breed" : "German Shepherd" }

Removing Content from RIAK

Here's a snippet of script to remove a content from Riak


let delete bucket key =
    let delete_url= sprintf "%s/riak/%s/%s" riakurl bucket key 
    client.DeleteAsync(delete_url)

delete "animals" "ace"

Getting Bucket Keys

To get all keys in a bucket

let get_keys bucket =
    let get_url = sprintf "%s/riak/%s?keys=true" riakurl bucket
    get_url |> client.GetStringAsync

let results = get_keys "animals"
printfn "%s" results.Result

The above script would return the following (reformatted for legibility purposes):

{"props":{"name":"animals",
          "allow_mult":false,
    "basic_quorum":false,
    "big_vclock":50,
    "chash_keyfun":{"mod":"riak_core_util",
                    "fun":"chash_std_keyfun"},
    "dw":"quorum",
    "last_write_wins":false,
    "linkfun":{"mod":"riak_kv_wm_link_walker",
               "fun":"mapreduce_linkfun"},
    "n_val":3,
    "notfound_ok":true,
    "old_vclock":86400,
    "postcommit":[],
    "pr":0,
    "precommit":[],
    "pw":0,
    "r":"quorum",
    "rw":"quorum",
    "small_vclock":50,
    "w":"quorum",
    "young_vclock":20},
    "keys":["ace","polly"]}

Retrieving Content from Riak

To retrieve content:

let get bucket key = 
    let get_url = sprintf "%s/riak/%s/%s/" riakurl bucket key
    get_url |> client.GetStringAsync

let results = get "animals" "ace"
printfn "%s" results.Result

Monday, March 25, 2013

Compare Tibco EMS Performance Under Load with Clojure

We have migrated our Tibco EMS infrastructure from the older Sun V490 servers to the newer HP ProLiant DL380 G7 servers. One of the concerns was how does the new servers perform under load compared with the older servers. I needed to create a load test that compared the performance of the two different types of servers. I turned to Clojure to help me put together this load test and here is the load test code:


Load Test Script in Clojure

; Leveraging Apache Commons to generate the random message
(def msg64 (RandomStringUtils/randomAlphanumeric 64))
(def msg128 (RandomStringUtils/randomAlphanumeric 128))
(def msg256 (RandomStringUtils/randomAlphanumeric 256))
(def msg64k (RandomStringUtils/randomAlphanumeric (* 64 1024)))
(def msg128k (RandomStringUtils/randomAlphanumeric (* 128 1024)))
(def msg256k (RandomStringUtils/randomAlphanumeric (* 256 1024)))
(def msg512k (RandomStringUtils/randomAlphanumeric (* 512 1024)))
(def msg1mb (RandomStringUtils/randomAlphanumeric (* 1024 1024)))
(def persistent-delivery (.intValue (javax.jms.DeliveryMode/PERSISTENT)))


; Send messages
(defn send-messages [server-url user password queue-name data iterations]
  (with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
            (.createQueueConnection user password))]
    (let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE) 
         queue   (.createQueue session  queue-name)]
      (with-open [sender (.createSender session queue)]
  (dotimes [_ iterations]
          (let [message (.createTextMessage session)]
            (.setJMSDeliveryMode message persistent-delivery)
            (.setText message data)
            (.send sender message)))))))
   

(def test-scenarios (list
 {:label "64 bytes" :msg msg64 :n 5000 }
 {:label "128 bytes" :msg msg128 :n 5000 }
 {:label "256 bytes" :msg msg256 :n 5000 }
 {:label "64KB bytes" :msg msg64k :n 2000 }
 {:label "128KB bytes" :msg msg128k :n 1000 }
 {:label "256KB bytes" :msg msg256k :n 1000 }
 {:label "512KB bytes" :msg msg512k :n 1000 }
 {:label "1MB bytes" :msg msg1mb :n 1000 }))

(doseq [my-test test-scenarios]
  (let [label (:label my-test)
       message (:msg my-test)
       n (:n my-test)]
    (println "\n\n")
 (println (str "Testing for " label " sized messages with n = " n))
 (time (send-messages server-url username password test-queue message n))))

Here's the message consumer part of the load test:


Message Consumer in Clojure

                
; Consume Queue Text messages asynchronously
(defn get-queue-text-messages [server-url user password queue-name process-message]
    (future
        (with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
                                   (.createQueueConnection user password))]
            (let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE)
                  queue (.createQueue session  queue-name)]
                (with-open [receiver (.createReceiver session queue)]             
                    (.start connection)
                    (loop [acc 0]                      
                        (process-message (.receive receiver) acc)
                        (recur (+ acc 1))))))))

; Dump just the message id      
(defn dump-message-id [message n]
    (println (str n)))

     
; Create function aliases with connection information embedded                   
(defn consume-messages [queue-name message-processor]
    (get-queue-text-messages  server-url username password queue-name message-processor))
 
; Start consuming messages asynchronously
(consume-messages queueName dump-message-id)     

Here are the test results:


Load Test Result on Sun V490

Message SizeNumber of MessagesProcess Time (ms)Throughput (msg/s)Throughput (MB/s)
64 5,000 2,772.84 1,803.21 0.11
128 5,000 1,943.23 2,573.04 0.31
256 5,000 1,899.24 2,632.63 0.64
64K 1,000 3,020.98 331.02 20.69
128K 1,000 4,414.00 226.55 28.32
256K 1,000 20,911.61 47.82 11.96
512K 1,000 39,213.23 25.50 12.75
1MB 1,000 80,337.55 12.45 12.45

Load Test Result on HP ProLiant DL380 G7

Message SizeNumber of MessagesProcess Time (ms)Throughput (msg/s)Throughput (MB/s)
64 5,000 1,340.22 3,730.74 0.23
128 5,000 1,345.18 3,716.99 0.45
256 5,000 1,040.59 4,804.95 1.17
64K 1,000 2,178.69 458.99 28.69
128K 1,000 3,553.02 281.45 35.18
256K 1,000 7,490.54 133.50 33.38
512K 1,000 35,078.26 28.51 14.25
1MB 1,000 77,076.51 12.97 12.97

From this test result, we can conclude that HP ProLiant DL380 G7 performed better than Sun V490 under load.

Monday, March 04, 2013

Retrieving VMware vSphere Performance Metrics Catalog with F#

vSphere collects a lot of performance metrics data on a VMware cluster. Our problem was that we cannot find a complete documentation on what are all those performance metrics data collected and what are the units of the collected metric data. We have asked VMware professional services engineer about this and his response was that VMware does not have the metrics data documented.

However, in Chapter 16 of the vSphere Web Services Programming Guide, it talked about the fact that you can retrieve the metrics catalog from vSphere itself. Although this document talks about retrieving the metrics catalog via vSphere Web Services SDK, you can also retrieve the metrics catalog with PowerCLI. Below is a F# script that retrieves vSphere metrics catalog:

#r @"C:\pkg\VMware\Infrastructure\vSphere PowerCLI\VMware.Vim.dll"

open System
open VMware.Vim
open System.Collections.Specialized

let client = new VimClient()
let service = client.Connect(serviceUrl)

// Must login to do anything - if you are getting null values, it means the session automatically timed out
client.Login(userId,password)

let printMetricCatalog () =
 // counterId max is arbitrarily set, I tried other values
 // and still produced the same number of returned metrics
    let counterIds = seq {1 .. 1000}
    let perfMgr = new PerformanceManager(client,client.ServiceContent.PerfManager)
    let metrics = perfMgr.QueryPerfCounter(counterIds |> Seq.toArray)

    let dumpAllMetricData (metric:PerfCounterInfo) = 
        printfn "-------------------------------------------------------------------\n"
        printfn "Key             : %i" metric.Key
        printfn "Level           : %A" metric.Level
        printfn "PerDeviceLevel  : %A" metric.PerDeviceLevel
        printfn "RollupType      : %A" metric.RollupType
        printfn "StatsType       : %A" metric.StatsType
        printfn "Group.Label     : %s" metric.GroupInfo.Label
        printfn "Group.Key       : %s" metric.GroupInfo.Key
        printfn "Group.Summary   : %s" metric.GroupInfo.Summary
        printfn "NameInfo.Label  : %s" metric.NameInfo.Label
        printfn "NameInfo.Key    : %s" metric.NameInfo.Key
        printfn "NameInfo.Summary: %s" metric.NameInfo.Summary
        printfn "UnitInfo.Label  : %s" metric.UnitInfo.Label
        printfn "UnitInfo.Key    : %s" metric.UnitInfo.Key
        printfn "UnitInfo.Summary: %s" metric.UnitInfo.Summary


    let dumpMetricData (metric:PerfCounterInfo) = printfn "\t%s (%s) - %A stat,  rollup: %A" 
                                                          (metric.NameInfo.Label) 
                                                          (metric.UnitInfo.Label) 
                                                          (metric.StatsType)
                                                          (metric.RollupType)

    // Break metrics catalog by metric groups
    let metricGroups = metrics |> Seq.map (fun metric -> metric.GroupInfo.Label) |> Set.ofSeq |> Set.toSeq

    // Dump short version of metrics catalog
    metricGroups
    |> Seq.iter (fun group ->  printfn "\n--------------------------------------------------"
                               printfn " Metric Group : %s" group
                               printfn "--------------------------------------------------"
                               metrics
                               |> Seq.filter (fun metric -> metric.GroupInfo.Label = group)
                               |> Seq.iter dumpMetricData)

    // Dump complete version of metrics catalog
    metrics |> Seq.iter dumpAllMetricData

printMetricCatalog()

Here's a dump of the metrics catalog reformatted to display as HTML tables:

VMware vSphere Metrics Catalog


CPU

MetricUnitStats TypeRollupDescription
CPU Capacity Contention Percent rate average Percent of time the VMs on this host are unable to run because they are contending for access to the physical CPU(s)
CPU Capacity Demand MHz absolute average The amount of CPU resources VMs on this host would use if there were no CPU contention or CPU limit
CPU Capacity Entitlement MHz absolute average CPU resources devoted by the ESX scheduler to virtual machines and resource pools
CPU Capacity Provisioned MHz absolute average Capacity in MHz of the physical CPU cores
CPU Capacity Usage MHz rate average CPU usage in MHz during the interval
CPU Core Count Contention Percent rate average Time the VM is ready to run, but is unable to run due to co-scheduling constraints
CPU Core Count Provisioned Number absolute average The number of physical cores provisioned to the entity
CPU Core Count Usage Number absolute average The number of virtual processors running on the host
Co-stop Millisecond delta summation Time the VM is ready to run, but is unable to due to co-scheduling constraints
Core Utilization Percent rate none CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Core Utilization Percent rate average CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Core Utilization Percent rate maximum CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Core Utilization Percent rate minimum CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Demand MHz absolute average The amount of CPU resources a VM would use if there were no CPU contention or CPU limit
Entitlement MHz absolute latest CPU resources devoted by the ESX scheduler
Idle Millisecond delta summation Total time that the CPU spent in an idle state
Latency Percent rate average Percent of time the VM is unable to run because it is contending for access to the physical CPU(s)
Max limited Millisecond delta summation Time the VM is ready to run, but is not run due to maxing out its CPU limit setting
Overlap Millisecond delta summation Time the VM was interrupted to perform system services on behalf of that VM or other VMs
Ready Millisecond delta summation Percentage of time that the virtual machine was ready, but could not get scheduled to run on the physical CPU
Reserved capacity MHz absolute average Total CPU capacity reserved by virtual machines
Run Millisecond delta summation Time the VM is scheduled to run
Swap wait Millisecond delta summation CPU time spent waiting for swap-in
System Millisecond delta summation Amount of time spent on system processes on each virtual CPU in the virtual machine
Total MHz rate average Total amount of CPU resources of all hosts in the cluster
Total capacity MHz absolute average Total CPU capacity reserved by and available for virtual machines
Usage Percent rate none CPU usage as a percentage during the interval
Usage Percent rate average CPU usage as a percentage during the interval
Usage Percent rate minimum CPU usage as a percentage during the interval
Usage Percent rate maximum CPU usage as a percentage during the interval
Usage in MHz MHz rate none CPU usage in megahertz during the interval
Usage in MHz MHz rate average CPU usage in megahertz during the interval
Usage in MHz MHz rate minimum CPU usage in megahertz during the interval
Usage in MHz MHz rate maximum CPU usage in megahertz during the interval
Used Millisecond delta summation Total CPU usage
Utilization Percent rate none CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Utilization Percent rate average CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Utilization Percent rate maximum CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Utilization Percent rate minimum CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Wait Millisecond delta summation Total CPU time spent in wait state
Worst case allocation MHz absolute latest Amount of CPU resources allocated to the virtual machine or resource pool based on the total cluster capacity and the resource configuration of the resource hierarchy

Cluster services

MetricUnitStats TypeRollupDescription
CPU fairness Number absolute latest Fairness of distributed CPU resource allocation
Current failover level Number absolute latest vSphere HA number of failures that can be tolerated
Effective CPU resources MHz rate average Total available CPU resources of all hosts within a cluster
Effective memory resources MB absolute average Total amount of machine memory of all hosts in the cluster that is available for use for virtual machine memory and overhead memory
Memory fairness Number absolute latest Aggregate available memory resources of all the hosts within a cluster

Datastore

MetricUnitStats TypeRollupDescription
Average read requests per second Number rate average Average number of read commands issued per second to the datastore during the collection interval
Average write requests per second Number rate average Average number of write commands issued per second to the datastore during the collection interval
Highest latency Millisecond absolute latest Highest latency value across all datastores used by the host
Read latency Millisecond absolute average The average time a read from the datastore takes
Read rate KBps rate average Rate of reading data from the datastore
Storage DRS datastore bytes read Number absolute latest Storage DRS datastore bytes read
Storage DRS datastore bytes written Number absolute latest Storage DRS datastore bytes written
Storage DRS datastore normalized read latency Number absolute latest Storage DRS datastore normalized read latency
Storage DRS datastore normalized write latency Number absolute latest Storage DRS datastore normalized write latency
Storage DRS datastore outstanding read requests Number absolute latest Storage DRS datastore outstanding read requests
Storage DRS datastore outstanding write requests Number absolute latest Storage DRS datastore outstanding write requests
Storage DRS datastore read I/O rate Number absolute latest Storage DRS datastore read I/O rate
Storage DRS datastore read workload metric Number absolute latest Storage DRS datastore metric for read workload model
Storage DRS datastore write I/O rate Number absolute latest Storage DRS datastore write I/O rate
Storage DRS datastore write workload metric Number absolute latest Storage DRS datastore metric for write workload model
Storage I/O Control aggregated IOPS Number absolute average Storage I/O Control aggregated IOPS
Storage I/O Control datastore maximum queue depth Number absolute latest Storage I/O Control datastore maximum queue depth
Storage I/O Control normalized latency Microsecond absolute average Storage I/O Control size-normalized I/O latency
Write latency Millisecond absolute average The average time a write to the datastore takes
Write rate KBps rate average Rate of writing data to the datastore
busResets Number delta summation busResets
commandsAborted Number delta summation commandsAborted
contention Millisecond absolute average contention
usage KBps absolute average usage

Disk

MetricUnitStats TypeRollupDescription
Average commands issued per second Number rate average Average number of SCSI commands issued per second during the collection interval
Average read requests per second Number rate average Average number of disk reads per second during the collection interval
Average write requests per second Number rate average Average number of disk writes per second during the collection interval
Bus resets Number delta summation Number of SCSI-bus reset commands issued during the collection interval
Capacity KB absolute latest Configured size of the datastore
Command latency Millisecond absolute average Average amount of time taken during the collection interval to process a SCSI command issued by the Guest OS to the virtual machine
Commands issued Number delta summation Number of SCSI commands issued during the collection interval
Commands terminated Number delta summation Number of SCSI commands terminated during the collection interval
Disk SCSI Reservation Conflicts Number delta summation Number of SCSI reservation conflicts for the LUN during the collection interval
Disk SCSI Reservation Conflicts % Percent absolute average Number of SCSI reservation conflicts for the LUN as a percent of total commands during the collection interval
Disk Throughput Contention Millisecond absolute average Average amount of time for an I/O operation to complete
Disk Throughput Usage KBps rate average Aggregated disk I/O rate
Highest latency Millisecond absolute latest Highest latency value across all disks used by the host
Kernel command latency Millisecond absolute average Average amount of time, in milliseconds, spent by VMkernel to process each SCSI command
Kernel read latency Millisecond absolute average Average amount of time, in milliseconds, spent by VMKernel to process each SCSI read command
Kernel write latency Millisecond absolute average Average amount of time, in milliseconds, spent by VMKernel to process each SCSI write command
Maximum queue depth Number absolute average Maximum queue depth
Overhead due to delta disk backings KB absolute latest Storage overhead of a virtual machine or a datastore due to delta disk backings
Physical device command latency Millisecond absolute average Average amount of time, in milliseconds, to complete a SCSI command from the physical device
Physical device read latency Millisecond absolute average Average amount of time, in milliseconds, to read from the physical device
Physical device write latency Millisecond absolute average Average amount of time, in milliseconds, to write to the physical device
Queue command latency Millisecond absolute average Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval
Queue read latency Millisecond absolute average Average amount of time spent in the VMkernel queue, per SCSI read command, during the collection interval
Queue write latency Millisecond absolute average Average amount time spent in the VMkernel queue, per SCSI write command, during the collection interval
Read latency Millisecond absolute average Average amount of time taken during the collection interval to process a SCSI read command issued from the Guest OS to the virtual machine
Read rate KBps rate average Average number of kilobytes read from the disk each second during the collection interval
Read requests Number delta summation Number of disk reads during the collection interval
Space actually used KB absolute latest Amount of space actually used by the virtual machine or the datastore
Space not shared KB absolute latest Amount of space associated exclusively with a virtual machine
Space potentially used KB absolute latest Amount of storage set aside for use by a datastore or a virtual machine
Usage KBps rate none Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Usage KBps rate average Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Usage KBps rate minimum Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Usage KBps rate maximum Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Write latency Millisecond absolute average Average amount of time taken during the collection interval to process a SCSI write command issued by the Guest OS to the virtual machine
Write rate KBps rate average Average number of kilobytes written to disk each second during the collection interval
Write requests Number delta summation Number of disk writes during the collection interval
contention Percent absolute average contention
provisioned KB absolute average provisioned
usage KB absolute average usage

Management agent

MetricUnitStats TypeRollupDescription
CPU usage MHz rate average Amount of Service Console CPU usage
Memory swap in KBps rate average Amount of memory that is swapped in for the Service Console
Memory swap out KBps rate average Amount of memory that is swapped out for the Service Console
Memory swap used KB absolute average Sum of the memory swapped by all powered-on virtual machines on the host
Memory used KB absolute average Amount of total configured memory that is available for use

Memory

MetricUnitStats TypeRollupDescription
Active KB absolute none Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active KB absolute average Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active KB absolute minimum Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active KB absolute maximum Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active write KB absolute average Amount of memory that is actively being written to by the VM
Balloon KB absolute none Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon KB absolute average Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon KB absolute minimum Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon KB absolute maximum Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon target KB absolute none Target value set by VMkernal for the virtual machine's memory balloon size
Balloon target KB absolute average Target value set by VMkernal for the virtual machine's memory balloon size
Balloon target KB absolute minimum Target value set by VMkernal for the virtual machine's memory balloon size
Balloon target KB absolute maximum Target value set by VMkernal for the virtual machine's memory balloon size
Compressed KB absolute average Amount of memory compressed by ESX
Compression rate KBps rate average Rate of memory compression for the VM
Consumed KB absolute none Amount of memory consumed by a virtual machine, host, or cluster
Consumed KB absolute average Amount of memory consumed by a virtual machine, host, or cluster
Consumed KB absolute minimum Amount of memory consumed by a virtual machine, host, or cluster
Consumed KB absolute maximum Amount of memory consumed by a virtual machine, host, or cluster
Decompression rate KBps rate average Rate of memory decompression for the VM
Entitlement KB absolute average Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
Granted KB absolute none Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Granted KB absolute average Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Granted KB absolute minimum Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Granted KB absolute maximum Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Heap KB absolute none VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap KB absolute average VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap KB absolute minimum VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap KB absolute maximum VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap free KB absolute none Free address space in the VMkernel's main heap
Heap free KB absolute average Free address space in the VMkernel's main heap
Heap free KB absolute minimum Free address space in the VMkernel's main heap
Heap free KB absolute maximum Free address space in the VMkernel's main heap
Host cache used for swapping KB absolute none Space used for caching swapped pages in the host cache
Host cache used for swapping KB absolute average Space used for caching swapped pages in the host cache
Host cache used for swapping KB absolute maximum Space used for caching swapped pages in the host cache
Host cache used for swapping KB absolute minimum Space used for caching swapped pages in the host cache
Latency Percent rate average Percentage of time the VM is waiting to access swapped or compressed memory
Low free threshold KB absolute average Threshold of free host physical memory below which ESX will begin reclaiming memory from VMs through ballooning and swapping
Memory Capacity Contention Percent rate average Percentage of time the VM is waiting to access swapped, compressed, or ballooned memory
Memory Capacity Entitlement KB absolute average Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
Memory Capacity Provisioned KB absolute average Total amount of memory configured for the VM
Memory Capacity Usable KB absolute average Amount of physical memory available for use by virtual machines on this host
Memory Capacity Usage KB absolute average Amount of physical memory actively used
Memory Consumed by VMs KB absolute average Amount of physical memory consumed by VMs on this host
Memory Consumed by userworlds KB absolute average Amount of physical memory consumed by userworlds on this host
Memory Reserved Capacity % Percent absolute average Percent of memory that has been reserved either through VMkernel use, by userworlds, or due to VM memory reservations
Memory saved by zipping KB absolute latest Memory (KB) saved due to memory zipping
Overhead KB absolute none Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead KB absolute average Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead KB absolute minimum Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead KB absolute maximum Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead touched KB absolute average Actively touched overhead memory (KB) reserved for use as the virtualization overhead for the VM
Reserved capacity MB absolute average Total amount of memory reservation used by powered-on virtual machines and vSphere services on the host
Reserved overhead KB absolute average Memory (KB) reserved for use as the virtualization overhead for the VM
Shared KB absolute none Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared KB absolute average Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared KB absolute minimum Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared KB absolute maximum Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared common KB absolute none Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
Shared common KB absolute average Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
Shared common KB absolute minimum Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
Shared common KB absolute maximum Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
State Number absolute latest One of four threshold levels representing the percentage of free memory on the host. The counter value determines swapping and ballooning behavior for memory reclamation.
Swap in KB absolute none Amount swapped-in to memory from disk
Swap in KB absolute average Amount swapped-in to memory from disk
Swap in KB absolute maximum Amount swapped-in to memory from disk
Swap in KB absolute minimum Amount swapped-in to memory from disk
Swap in from host cache KB absolute none Amount of memory swapped-in from host cache
Swap in from host cache KB absolute average Amount of memory swapped-in from host cache
Swap in from host cache KB absolute maximum Amount of memory swapped-in from host cache
Swap in from host cache KB absolute minimum Amount of memory swapped-in from host cache
Swap in rate KBps rate average Rate at which memory is swapped from disk into active memory during the interval
Swap in rate from host cache KBps rate average Rate at which memory is being swapped from host cache into active memory
Swap out KB absolute none Amount of memory swapped-out to disk
Swap out KB absolute average Amount of memory swapped-out to disk
Swap out KB absolute maximum Amount of memory swapped-out to disk
Swap out KB absolute minimum Amount of memory swapped-out to disk
Swap out rate KBps rate average Rate at which memory is being swapped from active memory to disk during the current interval
Swap out rate to host cache KBps rate average Rate at which memory is being swapped from active memory to host cache
Swap out to host cache KB absolute none Amount of memory swapped-out to host cache
Swap out to host cache KB absolute average Amount of memory swapped-out to host cache
Swap out to host cache KB absolute maximum Amount of memory swapped-out to host cache
Swap out to host cache KB absolute minimum Amount of memory swapped-out to host cache
Swap target KB absolute none Target size for the virtual machine swap file
Swap target KB absolute average Target size for the virtual machine swap file
Swap target KB absolute minimum Target size for the virtual machine swap file
Swap target KB absolute maximum Target size for the virtual machine swap file
Swap unreserved KB absolute none Amount of memory that is unreserved by swap
Swap unreserved KB absolute average Amount of memory that is unreserved by swap
Swap unreserved KB absolute minimum Amount of memory that is unreserved by swap
Swap unreserved KB absolute maximum Amount of memory that is unreserved by swap
Swap used KB absolute none Amount of memory that is used by swap
Swap used KB absolute average Amount of memory that is used by swap
Swap used KB absolute minimum Amount of memory that is used by swap
Swap used KB absolute maximum Amount of memory that is used by swap
Swapped KB absolute none Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Swapped KB absolute average Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Swapped KB absolute minimum Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Swapped KB absolute maximum Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Total MB absolute average Total amount of machine memory of all hosts in the cluster that is available for virtual machine memory (physical memory for use by the Guest OS) and virtual machine overhead memory
Total capacity MB absolute average Total amount of memory reservation used by and available for powered-on virtual machines and vSphere services on the host
Unreserved KB absolute none Amount of memory that is unreserved
Unreserved KB absolute average Amount of memory that is unreserved
Unreserved KB absolute minimum Amount of memory that is unreserved
Unreserved KB absolute maximum Amount of memory that is unreserved
Usage Percent absolute none Memory usage as percentage of total configured or available memory
Usage Percent absolute average Memory usage as percentage of total configured or available memory
Usage Percent absolute minimum Memory usage as percentage of total configured or available memory
Usage Percent absolute maximum Memory usage as percentage of total configured or available memory
Used by VMkernel KB absolute none Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Used by VMkernel KB absolute average Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Used by VMkernel KB absolute maximum Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Used by VMkernel KB absolute minimum Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Worst case allocation MB absolute latest Memory allocation as calculated by the VMkernel scheduler based on current estimated demand and reservation, limit, and shares policies set for all virtual machines and resource pools in the host or cluster
Zero KB absolute none Memory that contains 0s only
Zero KB absolute average Memory that contains 0s only
Zero KB absolute minimum Memory that contains 0s only
Zero KB absolute maximum Memory that contains 0s only
Zipped memory KB absolute latest Memory (KB) zipped
swapIn KB absolute none swapIn
swapIn KB absolute average swapIn
swapIn KB absolute minimum swapIn
swapIn KB absolute maximum swapIn
swapOut KB absolute none swapOut
swapOut KB absolute average swapOut
swapOut KB absolute minimum swapOut
swapOut KB absolute maximum swapOut
userworld KB absolute average userworld
userworld KB absolute average userworld
vm KB absolute average vm
vm KB absolute average vm
vmOvhd KB absolute average vmOvhd
vmOvrhd KB absolute average vmOvrhd
vmkOvrhd KB absolute average vmkOvrhd
vmkOvrhd KB absolute average vmkOvrhd

Network

MetricUnitStats TypeRollupDescription
Broadcast receives Number delta summation Number of broadcast packets received during the sampling interval
Broadcast transmits Number delta summation Number of broadcast packets transmitted during the sampling interval
Data receive rate KBps rate average Average rate at which data was received during the interval
Data receive rate KBps rate average Average amount of data received per second
Data transmit rate KBps rate average Average rate at which data was transmitted during the interval
Data transmit rate KBps rate average Average amount of data transmitted per second
Multicast receives Number delta summation Number of multicast packets received during the sampling interval
Multicast transmits Number delta summation Number of multicast packets transmitted during the sampling interval
Packet receive errors Number delta summation Number of packets with errors received during the sampling interval
Packet transmit errors Number delta summation Number of packets with errors transmitted during the sampling interval
Packets received Number delta summation Number of packets received during the interval
Packets transmitted Number delta summation Number of packets transmitted during the interval
Receive packets dropped Number delta summation Number of receives dropped
Transmit packets dropped Number delta summation Number of transmits dropped
Unknown protocol frames Number delta summation Number of frames with unknown protocol received during the sampling interval
Usage KBps rate none Network utilization (combined transmit-rates and receive-rates) during the interval
Usage KBps rate average Network utilization (combined transmit-rates and receive-rates) during the interval
Usage KBps rate minimum Network utilization (combined transmit-rates and receive-rates) during the interval
Usage KBps rate maximum Network utilization (combined transmit-rates and receive-rates) during the interval
pNic Packets Received and Transmitted per Second Number rate average Average rate of packets received and transmitted per second
pNic Throughput Provisioned KBps absolute average Provisioned pNic I/O Throughput
pNic Throughput Usable KBps absolute average Usable pNic I/O Throughput
pNic Throughput Usage for FT KBps rate average Average pNic I/O rate for FT
pNic Throughput Usage for NFS KBps rate average Average pNic I/O rate for NFS
pNic Throughput Usage for VMs KBps rate average Average pNic I/O rate for VMs
pNic Throughput Usage for VR KBps rate average Average pNic I/O rate for VR
pNic Throughput Usage for iSCSI KBps rate average Average pNic I/O rate for iSCSI
pNic Throughput Usage for vMotion KBps rate average Average pNic I/O rate for vMotion
vNic Throughput Contention Number delta summation Count of vNic packet drops
vNic Throughput Usage KBps rate average Average vNic I/O rate

Power

MetricUnitStats TypeRollupDescription
Cap Watt absolute average Maximum allowed power usage
Energy usage Joule delta summation Total energy used since last stats reset
Host Power Capacity Provisioned Percent absolute average Current power usage as a percentage of maximum allowed power
Host Power Capacity Usable Watt absolute average Current maximum allowed power usage
Power Capacity Usage Watt absolute average Current power usage
Usage Watt rate average Current power usage

Resource group CPU

MetricUnitStats TypeRollupDescription
Active (1 min. average) Percent absolute latest CPU active average over 1 minute
Active (1 min. peak) Percent absolute latest CPU active peak over 1 minute
Active (15 min. average) Percent absolute latest CPU active average over 15 minutes
Active (15 min. peak) Percent absolute latest CPU active peak over 15 minutes
Active (5 min. average) Percent absolute latest CPU active average over 5 minutes
Active (5 min. peak) Percent absolute latest CPU active peak over 5 minutes
Group CPU sample count Number absolute latest Group CPU sample count
Group CPU sample period Millisecond absolute latest Group CPU sample period
Running (1 min. average) Percent absolute latest CPU running average over 1 minute
Running (1 min. peak) Percent absolute latest CPU running peak over 1 minute
Running (15 min. average) Percent absolute latest CPU running average over 15 minutes
Running (15 min. peak) Percent absolute latest CPU running peak over 15 minutes
Running (5 min. average) Percent absolute latest CPU running average over 5 minutes
Running (5 min. peak) Percent absolute latest CPU running peak over 5 minutes
Throttled (1 min. average) Percent absolute latest Amount of CPU resources over the limit that were refused, average over 1 minute
Throttled (15 min. average) Percent absolute latest Amount of CPU resources over the limit that were refused, average over 15 minutes
Throttled (5 min. average) Percent absolute latest Amount of CPU resources over the limit that were refused, average over 5 minutes

Storage adapter

MetricUnitStats TypeRollupDescription
Average commands issued per second Number rate average Average number of commands issued per second by the storage adapter during the collection interval
Average read requests per second Number rate average Average number of read commands issued per second by the storage adapter during the collection interval
Average write requests per second Number rate average Average number of write commands issued per second by the storage adapter during the collection interval
Highest latency Millisecond absolute latest Highest latency value across all storage adapters used by the host
Read latency Millisecond absolute average The average time a read by the storage adapter takes
Read rate KBps rate average Rate of reading data by the storage adapter
Storage Adapter Number Queued Number absolute average The current number of I/Os that are waiting to be issued
Storage Adapter Outstanding I/Os Percent absolute average The percent of I/Os that have been issued but have not yet completed
Storage Adapter Outstanding I/Os Number absolute average The number of I/Os that have been issued but have not yet completed
Storage Adapter Queue Command Latency Millisecond absolute average Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval
Storage Adapter Queue Depth Number absolute average The maximum number of I/Os that can be outstanding at a given time
Storage Adapter Throughput Contention Millisecond absolute average Average amount of time for an I/O operation to complete
Storage Adapter Throughput Usage KBps rate average The storage adapter's I/O rate
Write latency Millisecond absolute average The average time a write by the storage adapter takes
Write rate KBps rate average Rate of writing data by the storage adapter

Storage path

MetricUnitStats TypeRollupDescription
Average commands issued per second Number rate average Average number of commands issued per second on the storage path during the collection interval
Average read requests per second Number rate average Average number of read commands issued per second on the storage path during the collection interval
Average write requests per second Number rate average Average number of write commands issued per second on the storage path during the collection interval
Highest latency Millisecond absolute latest Highest latency value across all storage paths used by the host
Read latency Millisecond absolute average The average time a read issued on the storage path takes
Read rate KBps rate average Rate of reading data on the storage path
Storage Path Bus Resets Number delta summation Number of SCSI-bus reset commands issued during the collection interval
Storage Path Command Aborts Number delta summation Number of SCSI commands aborted during the collection interval
Storage Path Throughput Contention Millisecond absolute average Average amount of time for an I/O operation to complete
Storage Path Throughput Usage KBps rate average Storage path I/O rate
Write latency Millisecond absolute average The average time a write issued on the storage path takes
Write rate KBps rate average Rate of writing data on the storage path

System

MetricUnitStats TypeRollupDescription
Disk usage Percent absolute latest Amount of disk space usage for each mount point
Heartbeat Number delta summation Number of heartbeats issued per virtual machine during the interval
Heartbeat Number absolute latest Number of heartbeats issued per virtual machine during the interval
OS Uptime Second absolute latest Total time elapsed, in seconds, since last operating system boot-up
Resource CPU active (1 min. average) Percent absolute latest CPU active average over 1 minute of the system resource group
Resource CPU active (5 min. average) Percent absolute latest CPU active average over 5 minutes of the system resource group
Resource CPU allocation maximum, in MHz MHz absolute latest CPU allocation limit, in MHz, of the system resource group
Resource CPU allocation minimum, in MHz MHz absolute latest CPU allocation reservation, in MHz, of the system resource group
Resource CPU allocation shares Number absolute latest CPU allocation shares of the system resource group
Resource CPU maximum limited (1 min.) Percent absolute latest CPU maximum limited over 1 minute of the system resource group
Resource CPU maximum limited (5 min.) Percent absolute latest CPU maximum limited over 5 minutes of the system resource group
Resource CPU running (1 min. average) Percent absolute latest CPU running average over 1 minute of the system resource group
Resource CPU running (5 min. average) Percent absolute latest CPU running average over 5 minutes of the system resource group
Resource CPU usage (Average) MHz rate average Amount of CPU used by the Service Console and other applications during the interval
Resource CPU usage (Maximum) MHz rate maximum Amount of CPU used by the Service Console and other applications during the interval
Resource CPU usage (Minimum) MHz rate minimum Amount of CPU used by the Service Console and other applications during the interval
Resource CPU usage (None) MHz rate none Amount of CPU used by the Service Console and other applications during the interval
Resource memory allocation maximum, in KB KB absolute latest Memory allocation limit, in KB, of the system resource group
Resource memory allocation minimum, in KB KB absolute latest Memory allocation reservation, in KB, of the system resource group
Resource memory allocation shares Number absolute latest Memory allocation shares of the system resource group
Resource memory mapped KB absolute latest Memory mapped by the system resource group
Resource memory overhead KB absolute latest Overhead memory consumed by the system resource group
Resource memory share saved KB absolute latest Memory saved due to sharing by the system resource group
Resource memory shared KB absolute latest Memory shared by the system resource group
Resource memory swapped KB absolute latest Memory swapped out by the system resource group
Resource memory touched KB absolute latest Memory touched by the system resource group
Resource memory zero KB absolute latest Zero filled memory used by the system resource group
Uptime Second absolute latest Total time elapsed, in seconds, since last system startup

Virtual disk

MetricUnitStats TypeRollupDescription
Average number of outstanding read requests Number absolute latest Average number of outstanding read requests to the virtual disk during the collection interval
Average number of outstanding write requests Number absolute latest Average number of outstanding write requests to the virtual disk during the collection interval
Average read requests per second Number rate average Average number of read commands issued per second to the virtual disk during the collection interval
Average write requests per second Number rate average Average number of write commands issued per second to the virtual disk during the collection interval
Read latency Millisecond absolute average The average time a read from the virtual disk takes
Read rate KBps rate average Rate of reading data from the virtual disk
Read workload metric Number absolute latest Storage DRS virtual disk metric for the read workload model
Virtual Disk Throughput Contention Millisecond absolute average Average amount of time for an I/O operation to complete
Virtual Disk Throughput Usage KBps rate average Virtual disk I/O rate
Write latency Millisecond absolute average The average time a write to the virtual disk takes
Write rate KBps rate average Rate of writing data to the virtual disk
Write workload metric Number absolute latest Storage DRS virtual disk metric for the write workload model
busResets Number delta summation busResets
commandsAborted Number delta summation commandsAborted

Virtual machine operations

MetricUnitStats TypeRollupDescription
Storage vMotion count Number absolute latest Number of migrations with Storage vMotion (datastore change operations for powered-on VMs)
VM clone count Number absolute latest Number of virtual machine clone operations
VM create count Number absolute latest Number of virtual machine create operations
VM datastore change count (non-powered-on VMs) Number absolute latest Number of datastore change operations for powered-off and suspended virtual machines
VM delete count Number absolute latest Number of virtual machine delete operations
VM guest reboot count Number absolute latest Number of virtual machine guest reboot operations
VM guest shutdown count Number absolute latest Number of virtual machine guest shutdown operations
VM host and datastore change count (non-powered-on VMs) Number absolute latest Number of host and datastore change operations for powered-off and suspended virtual machines
VM host change count (non-powered-on VMs) Number absolute latest Number of host change operations for powered-off and suspended VMs
VM power off count Number absolute latest Number of virtual machine power off operations
VM power on count Number absolute latest Number of virtual machine power on operations
VM reconfigure count Number absolute latest Number of virtual machine reconfigure operations
VM register count Number absolute latest Number of virtual machine register operations
VM reset count Number absolute latest Number of virtual machine reset operations
VM standby guest count Number absolute latest Number of virtual machine standby guest operations
VM suspend count Number absolute latest Number of virtual machine suspend operations
VM template deploy count Number absolute latest Number of virtual machine template deploy operations
VM unregister count Number absolute latest Number of virtual machine unregister operations
vMotion count Number absolute latest Number of migrations with vMotion (host change operations for powered-on VMs)

vCenter debugging information

MetricUnitStats TypeRollupDescription
Activation count Number absolute maximum Activation operations in vCenter
Activation count Number absolute minimum Activation operations in vCenter
Activation count Number absolute summation Activation operations in vCenter
Activation latency Millisecond absolute maximum The latency of an activation operation in vCenter
Activation latency Millisecond absolute minimum The latency of an activation operation in vCenter
Activation latency Millisecond absolute summation The latency of an activation operation in vCenter
Host sync count Number absolute maximum The number of host sync operations in vCenter
Host sync count Number absolute minimum The number of host sync operations in vCenter
Host sync count Number absolute summation The number of host sync operations in vCenter
Host sync latency Millisecond absolute maximum The latency of a host sync operation in vCenter
Host sync latency Millisecond absolute minimum The latency of a host sync operation in vCenter
Host sync latency Millisecond absolute summation The latency of a host sync operation in vCenter
Inventory statistics Number absolute maximum vCenter inventory statistics
Inventory statistics Number absolute minimum vCenter inventory statistics
Inventory statistics Number absolute summation vCenter inventory statistics
Locking statistics Number absolute maximum vCenter locking statistics
Locking statistics Number absolute minimum vCenter locking statistics
Locking statistics Number absolute summation vCenter locking statistics
Managed object reference statistics Number absolute maximum Managed object reference counts in vCenter
Managed object reference statistics Number absolute minimum Managed object reference counts in vCenter
Managed object reference statistics Number absolute summation Managed object reference counts in vCenter
Miscellaneous Number absolute maximum Miscellaneous statistics
Miscellaneous Number absolute minimum Miscellaneous statistics
Miscellaneous Number absolute summation Miscellaneous statistics
Scoreboard statistics Number absolute maximum Object counts in vCenter
Scoreboard statistics Number absolute minimum Object counts in vCenter
Scoreboard statistics Number absolute summation Object counts in vCenter
Session statistics Number absolute maximum The statistics of client sessions connected to vCenter
Session statistics Number absolute minimum The statistics of client sessions connected to vCenter
Session statistics Number absolute summation The statistics of client sessions connected to vCenter
System statistics Number absolute maximum The statistics of vCenter as a running system such as thread statistics and heap statistics
System statistics Number absolute minimum The statistics of vCenter as a running system such as thread statistics and heap statistics
System statistics Number absolute summation The statistics of vCenter as a running system such as thread statistics and heap statistics
vCenter LRO statistics Number absolute maximum vCenter LRO statistics
vCenter LRO statistics Number absolute minimum vCenter LRO statistics
vCenter LRO statistics Number absolute summation vCenter LRO statistics
vCenter service statistics Number absolute maximum vCenter service statistics such as events, alarms, and tasks
vCenter service statistics Number absolute minimum vCenter service statistics such as events, alarms, and tasks
vCenter service statistics Number absolute summation vCenter service statistics such as events, alarms, and tasks

vCenter resource usage information

MetricUnitStats TypeRollupDescription
CPU privileged Percent rate average CPU used by vCenter in privileged mode
CPU process Percent rate average Total CPU used by vCenter
CPU queue length Number absolute average Processor queue length on the system where vCenter is running
CPU system Percent rate average Total system CPU used on the system where vCenter in running
CPU user Percent rate average CPU used by vCenter in user mode
Context switch rate Number rate average Number of context switches per second on the system where vCenter is running
Disk bytes read rate Number rate average Number of bytes read from the disk per second on the system where vCenter is running
Disk bytes written rate Number rate average Number of bytes written to the disk per second on the system where vCenter is running
Disk queue length Number absolute average Disk queue length on the system where vCenter is running
Disk read rate Number rate average Number of disk reads per second on the system where vCenter is running
Disk write rate Number rate average Number of disk writes per second on the system where vCenter is running
Network queue length Number absolute average Network queue length on the system where vCenter is running
Network usage Percent rate average Total network bytes received and sent per second on the system where vCenter is running
Page fault rate Number rate average Number of page faults per second on the system where vCenter is running
Physical memory KB absolute average Physical memory used by vCenter
Pool non-paged bytes KB absolute average Memory pooled for non-paged bytes on the system where vCenter is running
Pool paged bytes KB absolute average Memory pooled for paged bytes on the system where vCenter is running
Process handles Number absolute average Handles used by vCenter
Process threads Number absolute average Number of threads used by vCenter
Received packet rate Number rate average Rate of the number of total packets received per second on the system where vCenter is running
Sent packet rate Number rate average Number of total packets sent per second on the system where vCenter is running
System call rate Number rate average Number of systems calls made per second on the system where vCenter is running
System threads Number absolute average Number of threads on the system where vCenter is running
Total packet rate Number rate average Number of total packets sent and received per second on the system where vCenter is running
Virtual memory KB absolute average Virtual memory used by vCenter

vSphere Replication

MetricUnitStats TypeRollupDescription
Replication Data Receive Rate KBps rate average Average amount of data received per second
Replication Data Transmit Rate KBps rate average Average amount of data transmitted per second
vSphere Replication VM Count Number absolute average Current Number of Replicated VMs