tag:blogger.com,1999:blog-182819362024-03-20T08:07:01.394-07:00John Liao's BlogJohn Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.comBlogger77125tag:blogger.com,1999:blog-18281936.post-59871669093001728942013-06-24T06:00:00.000-07:002013-06-24T06:00:01.793-07:00Riak CAP Tuning and F#<p>
Riak provides the ability to <a href="http://docs.basho.com/riak/1.1.4/tutorials/fast-track/Tunable-CAP-Controls-in-Riak/">tune CAP</a>. CAP, which stands for Consistency, Availability, and Partition tolerance, does not seem like controls that are tunable. These terms seem evoke images of binary choices, as in either you have it or you don't. CAP terms by itself is ambiguous in their definitions. I'm not the only one who feels that way as can be seen in Daniel Abadi's <a href="http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html">blog post</a>. For me, it was more helpful for me to think of tradeoffs as consistency latency (time needed to achieve eventual consistency), performance (read/write latency), and node failure tolerance (how many nodes can fail and still have a working cluster).</p>
<p>Riak exposes their CAP tuning controls via the named variables N, R, and W. These variables are defined as follows:</p>
<dl>
<dt><b>N</b></dt><dd>Number of nodes to replicated a piece of data</dd>
<dt><b>R</b></dt><dd>Number of nodes to read data to be considered success (read failure tolerance)</dd>
<dt><b>W</b></dt><dd>Number of nodes to write data to be considered write complete (write fault tolerance)</dd>
</dl>
<dl>
<p>In addition, Riak exposes these additional tuning controls:</p>
<dt><b>PR</b></dt><dd>Number of primary, non-fallback nodes that must return results for a successful read</dd>
<dt><b>PW</b></dt><dd>Number of primary, non-fallback nodes that must accept a write</dd>
<dt><b>DW</b></dt><dd>Number of nodes which have received an acknowledgement of the write from the storage backend</dd>
</dl>
<hr/>
<h3>Bucket Level CAP Controls in Riak</h3>
<p>
Here's an example on how to set bucket level CAP settings in Riak with CorrugatedIron:
</p>
<pre class="brush: fsharp">
// Get existing bucket properties
let properties = ciClient.GetBucketProperties("animals",true).Value
// Set # of nodes a write must ultimately replicate to
// This should be set at the creation of the bucket
properties.SetNVal(3u)
// Set number of nodes that must successfully written before successful write response
properties.SetWVal(2u)
// Set # of nodes required to read a value succesfully
properties.SetRVal(1u)
// Set primary read value
properties.SetPrVal(1u)
// Set primary write value
properties.SetPwVal(1u)
// Set durable write value
properties.SetDwVal(1u)
// Change bucket properties with these new CAP control values
ciClient.SetBucketProperties("animals",properties)
</pre>
<hr/>
<h3>Per Request CAP Controls in Riak</h3>
<p>Riak allows you to tune CAP controls at per request level:</p>
<pre class="brush: fsharp">
// Setting W & DW on puts
let options = new RiakPutOptions()
options.SetW(3u).SetDw(1u)
let data = new RiakObject("animals","toto",{nickname="Toto"; breed="Cairn Terrier"; score=5})
ciClient.Put(data,options)
// Get item with R value set to 2
ciClient.Get("animals","toto",1u).Value.GetObject<Animal>()
// Specify quorum
let getOptions = new RiakGetOptions()
getOptions.SetR("quorum")
// Need to convert IRiakClient to RiakClient in order to set RiakGetOptions
let client = ciClient :?> RiakClient
client.Get("animals","toto",getOptions).Value.GetObject<Animal>()
</pre>
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com29tag:blogger.com,1999:blog-18281936.post-2014664172860897762013-06-03T06:00:00.000-07:002013-06-03T06:00:02.388-07:00Riak MapReduce with F#<style>
code {background:#F8F8FF; font-weight:bold; font-size:larger; }
#mytable {
width: 700px;
padding: 0;
margin: 0;
}
caption {
padding: 0 0 5px 0;
width: 700px;
font: italic 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
text-align: right;
}
th {
font: bold 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #4f6b72;
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
border-top: 1px solid #C1DAD7;
letter-spacing: 2px;
text-transform: uppercase;
text-align: left;
padding: 6px 6px 6px 12px;
background: #CAE8EA url(images/bg_header.jpg) no-repeat;
}
th.nobg {
border-top: 0;
border-left: 0;
border-right: 1px solid #C1DAD7;
background: none;
}
td {
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
background: #fff;
padding: 6px 6px 6px 12px;
color: #4f6b72;
}
td.alt {
background: #F5FAFA;
color: #797268;
}
th.spec {
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #fff url(images/bullet1.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
th.specalt {http://www.blogger.com/post-edit.g?blogID=18281936&postID=4559158457660466923
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #f5fafa url(images/bullet2.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #797268;
}
</style>
<p>
</p>
Lately, I have been reading the book
<a href="http://www.amazon.com/exec/obidos/ASIN/159420411X/techie2biz-20">
Signals and Noise</a> by Nate Silver. His book references an IBM
<a href="http://www-01.ibm.com/software/data/bigdata/">webpage</a>
that claims the world is creating 2.5 quintillion (10<sup>18</sup>) bytes of data a day and that 90% of the data that currently exists the world was created in the past two years. Combine this fact with the rise in
<a href="http://www.intel.com/content/www/us/en/high-performance-computing/high-performance-xeon-phi-coprocessor-brief.html">
multicore processors,</a> would explain the surging interest in functional programming and NoSQL and MapReduce frameworks. There is just too much data to be handled by a single machine alone</p>
<p>Continuing the journey in the book
<a href="http://www.amazon.com/exec/obidos/ASIN/1934356921/techie2biz-20">
Seven Databases in Seven Weeks</a> as I explore MapReduce. MapReduce framework is popularized by Google's <a href="http://research.google.com/archive/mapreduce.html">seminal paper</a>, but the genesis of that framework started with ideas in functional programming languages. After all, if functional programming is intended to help one run code on multiple processors on a single server, it's a logical conclusion to extend that computing from multiple processors on a single server to multiple servers. However, it's the recent technology factors and the data tsunami that I believe encourages people to look at functional programming and NoSQL databases.</p>
<p>What I would like to see in the future is that MapReduce capability is integrated with functional programming in such a way that calling a map/reduce function on a typed collection that recognizes that the typed collection is backed by a NoSQL database that it seamlessly spawns the function to the NoSQL stores in a MapReduce fashion.</p>
<p>I want to do something like the following:</p>
<pre class="brush: fsharp">
Seq.map myfunc myRiakCollection
</pre>
<p>and the compiler/runtime will automatically convert <code>myfunc</code> into an appropriate form to run on all the data nodes and returns the results. But since we're not there yet, we need to do a little more work.</p>
<p>Some additional items to note, <a href="http://riakhandbook.com/">Riak Handbook</a> warns that:</p>
<cite>
MapReduce in Riak is not exactly the same as Hadoop, where you can practically analyze
an infinite amount of data. In Riak, data is pulled out of the system to be
analyzed. That alone sets the boundaries of what's possible.
</cite>
<p>This is further reinforced by this following <a href="http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-May/004231.html">mailing list entry</a> in a response from Sean Cribbs, developer advocate from Basho:</p>
<cite>
...Riak's MapReduce is designed for low-latency queries, but it's also designed for "targeted" queries where you don't slurp the entire contents of a bucket from disk. Structure your data so that you can know -- or deterministically guess -- what keys should be involved in your query. Then you won't be trying to fetch and filter all of the data all the time.
</cite>
<p>I wanted to point this out as this goes against my mental model of how MapReduce should work.</P>
<hr/>
<h3>MapReduce with CorrugatedIron</h3>
<p>In this particular example, I will use the Javascript example from the book. First, we'll define some helper functions that provides some syntatic sugar in working with the CorrugatedIron's fluent interface:</p>
<pre class="brush: fsharp">
// Helper function for CorrugatedIron's job inputs for bucket/key pairs only.
let setinputs data (q:RiakMapReduceQuery) =
let inputs = new RiakBucketKeyInput()
data
|> List.iter( fun (bucket,key) -> inputs.AddBucketKey(bucket,key))
q.Inputs(inputs)
// Helper function mapping Javascript functions
let mapjs src keep (q:RiakMapReduceQuery) =
q.MapJs(fun m -> m.Source(src).Keep(keep) |> ignore)
// Helper function for mapping stored functions
let mapsf bucket func keep (q:RiakMapReduceQuery) =
q.MapJs(fun m -> m.BucketKey(bucket,func).Keep(keep) |> ignore)
// Helper function for mapping builtin functions
let mapbuiltins func keep (q:RiakMapReduceQuery) =
q.MapJs(fun m -> m.Name(func).Keep(true) |> ignore)
// Getting results from CorrugatedIron seems convoluted to me
// Helper function is used to extract the data to a simpler form
let getResults (results:RiakResult<RiakMapReduceResult>) =
let getText raw = raw |> System.Text.Encoding.Default.GetString
results.Value.PhaseResults
|> Seq.map (fun phase -> phase.Values |> Seq.map getText )
|> Seq.concat
// Helper function for Reduce operations in Javascript
let reducejs src keep (q:RiakMapReduceQuery) =
q.ReduceJs(fun m -> m.Source(src).Keep(keep) |> ignore)
// Helper function for link walking with mapreduce
let linkphase bucket keep (q:RiakMapReduceQuery) =
q.Link(fun l -> l.Keep(keep).Bucket(bucket) |> ignore)
</pre>
<p>Here's the script to perform the MapReduce in CorrugatedIrons. In the following snippet, <code>ciClient</code> refers to CorrugatedIron's <code>RiakClient</code> object, in the previous blogs, <code>ciClient</code> was just <code>client</code>:</p>
<pre class="brush: fsharp">
// Mapreduce example from the book
let src ="function(v) {
/* From the Riak object, pull data and parse it as JSON */
var parsed_data = JSON.parse(v.values[0].data);
var data = {};
/* Key capacity number by room style string */
data[parsed_data.style] = parsed_data.capacity;
return [data];
}"
let rooms = [("rooms","101");("rooms","102");("rooms","103");]
let query = new RiakMapReduceQuery()
|> setinputs rooms
|> mapjs src true
ciClient.MapReduce(query)
|> getResults
</pre>
<p>Results:</p>
<pre class="brush: plain">
seq ["[{"suite":3}]"; "[{"king":1}]"; "[{"double":4}]"]
</pre>
<hr/>
<h3>MapReduce with REST API</h3>
<p>It's not all that hard to do MapReduce with ASP.NET MVC REST API although your json script becomes more complex. In the following example, I had to escape the double quote(") because, alas, I'm still using F# 2.0. If you're using F# 3.0, you can use the new triple quote string syntax to wrap the MapReduce script.</p>
<pre class="brush: fsharp">
// This helper function will be used in all the later examples with REST API
let mapred script =
let post_url = sprintf "%s/mapred" riakurl
let content = new StringContent(script)
content.Headers.ContentType.MediaType <- "application/json"
let response = restClient.PostAsync(post_url,content)
response.Wait()
let results = response.Result.Content.ReadAsStringAsync()
results
let mapreduceScript = "{
\"inputs\":[
[\"rooms\",\"101\"],[\"rooms\",\"102\"],[\"rooms\",\"103\"]
],
\"query\":[
{\"map\":{
\"language\":\"javascript\",
\"source\":
\"function(v) {
/* From the Riak object, pull data and parse it as JSON */
var parsed_data = JSON.parse(v.values[0].data);
var data = {};
/* Key capacity number by room style string */
data[parsed_data.style] = parsed_data.capacity;
return [data];
}\"
}}
]
}"
mapred mapreduceScript
</pre>
<p>Results:</p>
<pre class="brush: plain">
val it : Task<string> =
System.Threading.Tasks.Task`1[System.String]
{AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 2;
IsCanceled = false;
IsCompleted = true;
IsFaulted = false;
Result = "[{"king":1},{"suite":3},{"double":4}]";
Status = RanToCompletion;}
</pre>
<hr/>
<h3>Stored Functions with CorrugatedIron</h3>
<p>I can store custom Javascript mapreduce functions in a specific bucket/key and call it for use:</p>
<pre class="brush: fsharp">
let myfunc = "
function(v) {
var parsed_data = JSON.parse(v.values[0].data);
var data = {};
data[parsed_data.style] = parsed_data.capacity;
return [data];
}"
// Store the map function in a bucket value
let putResults = new RiakObject("my_functions","map_capacity", myfunc)
|> ciClient.Put
// Querying using stored map function
let rooms = [("rooms","101");("rooms","102");("rooms","103");("rooms","104");]
let query = new RiakMapReduceQuery()
|> setinputs rooms
|> mapsf "my_functions" "map_capacity" true
ciClient.MapReduce(query)
|> getResults
</pre>
<p>Results:</p>
<pre class="brush: plain">
seq ["[{"suite":3}]"; "[{"king":1}]"; "[{"double":4}]"; "[{"suite":7}]"]
</pre>
<hr/>
<h3>Stored Functions with REST API</h3>
<p>With REST API, reusing the <code>mapred</code> function defined previously:</p>
<pre class="brush: fsharp">
let storedFuncScript = "
{
\"inputs\":[
[\"rooms\",\"101\"],[\"rooms\",\"102\"],[\"rooms\",\"103\"],[\"rooms\",\"104\"]
],
\"query\":[
{\"map\":{
\"language\":\"javascript\",
\"bucket\":\"my_functions\",
\"key\":\"map_capacity\"
}}
]
}
"
let savesf bucket key myfunc =
let post_url = sprintf "%s/riak/%s/%s" riakurl bucket key
let content = new StringContent(myfunc)
content.Headers.ContentType.MediaType <- "application/json"
restClient.PostAsync(post_url,content)
// Store the map function in a bucket value
savesf "my_functions" "map_capacity" myfunc
mapred storedFuncScript
</pre>
<p>Results:</p>
<pre class="brush: plain">
val it : TaskTask<string> =
System.Threading.Tasks.Task`1[System.String]
{AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 5;
IsCanceled = false;
IsCompleted = true;
IsFaulted = false;
Result = "[{"suite":3},{"double":4},{"king":1},{"suite":7}]";
Status = RanToCompletion;}
</pre>
<hr/>
<h3>Built-in Functions with CorrugatedIron</h3>
<p>Riak has some built-in mapreduce javascript functions. I couldn't find any reference documentation on it, the closes I came is the actual Javascript code in github: <a href="https://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js"></a>. Here are the list of builtin Javascript functions from that source code:</p>
<ul>
<li><code>Riak.getClassName(obj)</code></li>
<li><code>Riak.filterNotFound(values)</code></li>
<li><code>Riak.mapValues(value,keyData,arg)</code></li>
<li><code>Riak.mapValuesJson(value,keyData,arg)</code></li>
<li><code>Riak.mapByFields(value,keyData,fields)</code></li>
<li><code>Riak.reduceSum(values,arg)</code></li>
<li><code>Riak.reduceMin(values,arg)</code></li>
<li><code>Riak.reduceMax(values,arg)</code></li>
<li><code>Riak.reduceSort(value,arg)</code></li>
<li><code>Riak.reduceNumericSort(value, arg)</code></li>
<li><code>Riak.reduceLimit(value, arg)</code></li>
<li><code>Riak.reduceSlice(value, arg)</code></li></ul>
<p>In the following example, we're only going to use <code>Riak.mapValuesJson()</code>:</p>
<pre class="brush: fsharp">
let rooms = [("rooms","101");("rooms","102");("rooms","103");("rooms","104");]
let query = new RiakMapReduceQuery()
|> setinputs rooms
|> mapbuiltins "Riak.mapValuesJson" true
let results = ciClient.MapReduce(query)
// Collect the results and flatten the nested collections
results.Value.PhaseResults
|> Seq.map (fun x -> x.GetObjects<IEnumerable<Room>>())
|> Seq.concat
|> Seq.concat
</pre>
<p>Results:</p>
<pre class="brush: plain">
val it : seq<Room> =
seq [{style = "double"; capacity = 4;};
{style = "suite"; capacity = 3;};
{style = "suite"; capacity = 7;};
{style = "king"; capacity = 1;}]
</pre>
<hr/>
<h3>Built-in Functions with CorrugatedIron</h3>
<p>Calling built-in functions with REST API:</p>
<pre class="brush: fsharp">
let builtinFunc = "
{
\"inputs\":[
[\"rooms\",\"101\"],[\"rooms\",\"102\"],[\"rooms\",\"103\"]
],
\"query\":[
{\"map\":{
\"language\":\"javascript\",
\"name\":\"Riak.mapValuesJson\"
}}
]
}
"
mapred builtinFunc
</pre>
<p>Results:</p>
<pre class="brush: plain">
val it : Task<string> =
System.Threading.Tasks.Task`1[System.String]
{AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 6;
IsCanceled = false;
IsCompleted = true;
IsFaulted = false;
Result = "[{"style":"double","capacity":4},
{"style":"king","capacity":1},
{"style":"suite","capacity":3}]";
Status = RanToCompletion;}
</pre>
<hr/>
<h3>Reducing with CorrugatedIron</h3>
<p>Reduce operations with CorrguatedIron:</p>
<pre class="brush: fsharp">
let reduceScript = "
function(v) {
var totals = {};
for (var i in v) {
for(var style in v[i]) {
if (totals[style]) totals[style] += v[i][style];
else totals[style] = v[i][style];
}
}
return [totals];
}"
let query = (new RiakMapReduceQuery()).Inputs("rooms")
|> mapsf "my_functions" "map_capacity" false
|> reducejs reduceScript true
ciClient.MapReduce(query)
|> getResults
</pre>
<p>Results:</p>
<pre class="brush: plain">
seq["[{"queen":9051,"king":9189,"single":8811,"double":8629,"suite":9231}]"]
</pre>
<h3>Reduce operations with REST API</h3>
<p>MapReduce with REST API:</p>
<pre class="brush: fsharp">
let myReduceScript = "
{
\"inputs\":\"rooms\",
\"query\":[
{\"map\":{
\"language\":\"javascript\",
\"bucket\":\"my_functions\",
\"key\":\"map_capacity\"
}},
{\"reduce\":{
\"language\":\"javascript\",
\"source\":
\"function(v) {
var totals = {};
for (var i in v) {
for(var style in v[i]) {
if( totals[style] ) totals[style] += v[i][style];
else totals[style] = v[i][style];
}
}
return [totals];
}\"
}}
]
}"
mapred myReduceScript
</pre>
<p>Results:</p>
<pre class="brush: plain">
val it : Task<string> =
System.Threading.Tasks.Task`1[System.String]
{AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 7;
IsCanceled = false;
IsCompleted = true;
IsFaulted = false;
Result = "[{"suite":9231,"king":9189,"double":8629,"queen":9051,"single":8811}]";
Status = RanToCompletion;}
</pre>
<hr/>
<h3>Key Filters with CorrugatedIron</h3>
<p>Key filters is used to reduce the input set before the map/reduce operations:</p>
<pre class="brush: fsharp">
let reduceScript = "
function(v) {
var totals = {};
for (var i in v) {
for(var style in v[i]) {
if (totals[style]) totals[style] += v[i][style];
else totals[style] = v[i][style];
}
}
return [totals];
}"
let keyFilter = new Action<RiakFluentKeyFilter>(fun x -> x.StringToInt().LessThan(1000) |> ignore)
let query = (new RiakMapReduceQuery()).Inputs("rooms").Filter(keyFilter)
|> mapsf "my_functions" "map_capacity" false
|> reducejs reduceScript true
ciClient.MapReduce(query)
|> getResults
</pre>
<p>Results:</p>
<pre class="brush: plain">
val it : seq<string> =
seq ["[{"queen":780,"suite":968,"king":838,"single":791,"double":758}]"]
</pre>
<hr/>
<h3>Key Filters with REST API</h3>
<p>Adding key filters with REST API:</p>
<pre class="brush: fsharp">
let keyFilterScript = "
{
\"inputs\":{
\"bucket\":\"rooms\",
\"key_filters\":[[\"string_to_int\"], [\"less_than\", 1000]]},
\"query\":[
{\"map\":{
\"language\":\"javascript\",
\"bucket\":\"my_functions\",
\"key\":\"map_capacity\"
}},
{\"reduce\":{
\"language\":\"javascript\",
\"source\":
\"function(v) {
var totals = {};
for (var i in v) {
for(var style in v[i]) {
if( totals[style] ) totals[style] += v[i][style];
else totals[style] = v[i][style];
}
}
return [totals];
}\"
}}
]
}"
mapred keyFilterScript
</pre>
<p>Results:</p>
<pre class="brush: plain">
val it : Task<string> =
System.Threading.Tasks.Task`1[System.String]
{AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 10;
IsCanceled = false;
IsCompleted = true;
IsFaulted = false;
Result = "[{"queen":780,"king":838,"suite":968,"single":791,"double":758}]";
Status = RanToCompletion;}
</pre>
<hr/>
<h3>MapReduce Link Walking with CorrugatedIron</h3>
<p>In my previous blog, I talked about the fact I couldn't figure out how to specify the <code>keep</code> flag in the link walk examples. With map reduce link walking in CorrugatedIron, I now can now specify the <code>keep</code> flag for each of the link-map-reduce phases:</p>
<pre class="brush: fsharp">
let keyFilter = new Action<RiakFluentKeyFilter>(fun x -> x.Equal("2") |> ignore)
let query = (new RiakMapReduceQuery()).Inputs("cages").Filter(keyFilter)
|> linkphase "animals" false
|> mapjs "function(v) { return [v]; }" true
ciClient.MapReduce(query)
|> getResults
</pre>
<p>Results:</p>
<pre class="brush: plain">
val it : seq<seq<string>> =
seq
[seq [];
seq ["[{
"bucket":"animals",
"key":"ace",
"vclock":"a85hYGBgymDKBVIcrFrhrwI5O5wzmBKZ8lgZlj/hPM0HlVLm/PczkDP1FlRqHUgqCwA=",
"values":
[{"metadata":
{"X-Riak-VTag":"4XZxrmgvKzan8X4FKz5hti",
"content-type":"application/json",
"index":[],
"X-Riak-Last-Modified":"Thu, 16 May 2013 23:15:58 GMT"},
"data":"{\"nickname\":\"The Wonder Dog\",\"breed\":\"German Shepherd\"}"}]}]"]]
</pre>
<hr/>
<h3>MapReduce Link Walking with REST API</h3>
<p>With REST API:</p>
<pre class="brush: fsharp">
// Code for Link Walking with MapReduce
let mrLinkWalk = "
{
\"inputs\":{
\"bucket\":\"cages\",
\"key_filters\":[[\"eq\", \"2\"]]
},
\"query\":[
{\"link\":{
\"bucket\":\"animals\",
\"keep\":false
}},
{\"map\":{
\"language\":\"javascript\",
\"source\":
\"function(v) { return [v]; }\"
}}
]
}
"
mapred mrLinkWalk
</pre>
<p>Results:</p>
<pre class="brush: plain">
val it : Task<string> =
System.Threading.Tasks.Task`1[System.String]
{AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 13;
IsCanceled = false;
IsCompleted = true;
IsFaulted = false;
Result = "[{
"bucket":"animals",
"key":"ace",
"vclock":"a85hYGBgymDKBVIcrFrhrwI5O5wzmBKZ8lgZlj/hPM0HlVLm/PczkDP1FlRqHUgqCwA=",
"values":[{
"metadata":{
"X-Riak-VTag":"4XZxrmgvKzan8X4FKz5hti",
"content-type": "application/json",
"index":[],
"X-Riak-Last-Modified":"Thu, 16 May 2013 23:15:58 GMT"},
"data":"{\"nickname\":\"The Wonder Dog\",\"breed\":\"German Shepherd\"}"}]}]";
Status = RanToCompletion;}
</pre>
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-84206418152818981582013-05-12T21:00:00.000-07:002013-05-12T21:00:12.543-07:00Riak Links and Link Walking with F# and CorrugatedIron<style>
code {background:#F8F8FF; font-weight:bold; font-size:larger; }
#mytable {
width: 700px;
padding: 0;
margin: 0;
}
caption {
padding: 0 0 5px 0;
width: 700px;
font: italic 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
text-align: right;
}
th {
font: bold 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #4f6b72;
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
border-top: 1px solid #C1DAD7;
letter-spacing: 2px;
text-transform: uppercase;
text-align: left;
padding: 6px 6px 6px 12px;
background: #CAE8EA url(images/bg_header.jpg) no-repeat;
}
th.nobg {
border-top: 0;
border-left: 0;
border-right: 1px solid #C1DAD7;
background: none;
}
td {
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
background: #fff;
padding: 6px 6px 6px 12px;
color: #4f6b72;
}
td.alt {
background: #F5FAFA;
color: #797268;
}
th.spec {
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #fff url(images/bullet1.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
th.specalt {http://www.blogger.com/post-edit.g?blogID=18281936&postID=4559158457660466923
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #f5fafa url(images/bullet2.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #797268;
}
</style>
<p>Continuing my journey through the book <a href="http://www.amazon.com/exec/obidos/ASIN/1934356921/techie2biz-20">Seven Databases in Seven Weeks</a>, I explore links and link walking in this blog post.
Riak has the ability to establish one-way relationship between entries via <a href="http://docs.basho.com/riak/1.1.4/references/appendices/concepts/Links/">Links</a>, providing some of the capabilities of a graph database (Riak documentation calls it a lightweight graph database). <a href="http://docs.basho.com/riak/1.1.4/references/appendices/comparisons/Riak-Compared-to-Neo4j/">Riak documentation</a> hints that links should be kept low, on the order of dozens, not thousands. Using the Twitter example from <a href="http://riakhandbook.com">Riak Handbook</a>, you can probably easily find out who Don Syme follows and who does those people follows, etc. But it would be difficult to find all the people that follows Don Syme using Riak's link capability.</p>
<hr/>
<h3>Adding Links</h3>
<p>
Here's how you would add links with CorrugatedIron library:</p>
<pre class="brush: fsharp">
type Cage = { room : int }
(*
Linking cage 1 to polly via contains, equivalent to doing the following
curl -X PUT http://localhost:8098/riak/cages/1 \
-H "Content-Type: application/json" \
-H "Link: </riak/animals/polly>; riaktag=\"contains\"" \
-d '{"room" : 101}'
*)
client.Put(
let cage = new RiakObject("cages","1",{room=101})
cage.LinkTo("animals","polly","contains")
cage)
(*
Putting ace in cage 2 and setting cage 2 next to cage 1.
Equivalent to the following sample code:
curl -X PUT http://localhost:8091/riak/cages/2 \
-H "Content-Type: application/json" \
-H "Link:</riak/animals/ace>;riaktag=\"contains\",</riak/cages/1>;riaktag=\"next_to\"" \
-d '{"room" : 101}'
*)
// Adding more than one link
client.Put(
let cage = new RiakObject("cages","2",{room=101})
[("animals","ace","contains");("cages","1","next_to")]
|> List.iter(fun (bucket,key,tag) -> cage.LinkTo(bucket,key,tag))
cage)
</pre>
<pre class="brush: plain">
</pre>
<hr/>
<h3>Link Walking</h3>
<p>With CorrugatedIron, I can use the API to perform link walking and as an extra bonus, I don't have to worry about extracting data from the multipart/mixed mime types. CorrugateIron library takes care of all of that for us.</p>
<pre class="brush: fsharp">
(*
Link walking, equivalent to
curl http://riakhost1:8098/riak/cages/1/_,_,_
or in the new version:
curl http://riakhost1:8098/buckets/cages/keys/1/_,_,_
*)
let results = client.WalkLinks(new RiakObject("cages","1"),
[|new RiakLink(null,null,null)|])
// Dump results, which returns a list of RiakObject(s)
results.Value
// Since in this particular case, we have only one result, we can do the following
// and get back polly
results.Value.[0].GetObject<Animal>()
</pre>
<p>Here is the output result for <code>results.Value</code>:</p>
<pre class="brush: plain">
// results.Value
val it : IList<RiakObject> =
seq
[CorrugatedIron.Models.RiakObject
{BinIndexes = dict [];
Bucket = "animals";
CharSet = null;
ContentEncoding = null;
ContentType = "application/json";
HasChanged = false;
IntIndexes = dict [];
Key = "polly";
LastModified = 1359758822u;
LastModifiedUsec = 523420u;
Links = seq [];
Siblings = seq [];
UserMetaData = dict [];
VTag = "2BTveSKTYDNOZNCiOxyryw";
VTags = seq ["2BTveSKTYDNOZNCiOxyryw"];
Value = [|123uy; 32uy; 34uy; 110uy; 105uy; 99uy; 107uy; 110uy; 97uy;
109uy; 101uy; 34uy; 32uy; 58uy; 32uy; 34uy; 83uy; 119uy;
101uy; 101uy; 116uy; 32uy; 80uy; 111uy; 108uy; 108uy; 121uy;
32uy; 80uy; 117uy; 114uy; 101uy; 98uy; 114uy; 101uy; 100uy;
34uy; 32uy; 44uy; 32uy; 34uy; 98uy; 114uy; 101uy; 101uy;
100uy; 34uy; 32uy; 58uy; 32uy; 34uy; 80uy; 117uy; 114uy;
101uy; 98uy; 114uy; 101uy; 100uy; 34uy; 32uy; 125uy|];
VectorClock = [|107uy; 206uy; 97uy; 96uy; 96uy; 96uy; 204uy; 96uy;
202uy; 5uy; 82uy; 28uy; 169uy; 111uy; 239uy; 241uy;
7uy; 114uy; 230uy; 184uy; 103uy; 48uy; 37uy; 50uy;
230uy; 177uy; 50uy; 60uy; 59uy; 216uy; 112uy; 138uy;
47uy; 11uy; 0uy|];}]
// results.Value.[0].GetObject<Animal>()
val it : Animal = {nickname = "Sweet Polly Purebred";
breed = "Purebred";}
</pre>
<p>There seems to be some limitations with link walking using CorrugatedIron library. There is a <code>WalkLinks()</code> method as part <code>RiakClient</code>, but the input is expecting <code>RiakLink</code> object which has bucket,key, tag as arguments. The link spec is expecting bucket, tag, and keep flag as arguments. I do notice that in the Riak documentation that Link Walking is not available as part of the Protocol Buffers Client (PBC) API, so I'm guessing the <code>WalkLinks()</code> method in CorrugatedIron is either using HTTP protocol or a modified usage of MapReduce. Since link walking is a special case of MapReduce querying, it may not matter much that CorrugatedIron has some limitations on link walking. One other issue with CorrugatedIron and link walking is that when I add multiple links with the same tag and try to link walking with CorrugatedIron, I do not get back all the links, I only get one of the links. In order to follow the examples in the book <a href="http://www.amazon.com/exec/obidos/ASIN/1934356921/techie2biz-20">Seven Databases in Seven Weeks</a>, I can fall back to using ASP.NET MVC Rest API.</p>
<p>Link walking using CorrugatedIron library:</p>
<pre class="brush: fsharp">
// Link walking, equivalent to
// curl http://localhost:8098/riak/cages/2/animals,_,_
client.WalkLinks(new RiakObject("cages","2"),
[|new RiakLink("animals",null,null)|])
// curl http://localhost:8098/riak/cages/2/_,next_to,0/animals,_,_
client.WalkLinks(new RiakObject("cages","2"),
[|new RiakLink(null,null,"next_to");
new RiakLink("animals",null,null);|])
// I can't seem to specify the keep flag with CorrugatedIron, which keeps
// intermediate results as you walk beyond primary links
// curl http://localhost:8091/riak/cages/2/_,next_to,1/_,_,_
</pre>
<p>The book Seven Databases in Seven Weeks is still using the old format for HTTP Link Walking. The <a href="http://docs.basho.com/riak/latest/references/apis/http/HTTP-Link-Walking/#Example">new format</a> is as follows:
<pre class="brush: plain">
GET /riak/bucket/key/[bucket],[tag],[keep] # Old format
GET /buckets/bucket/keys/key/[bucket],[tag],[keep] # New format
</pre>
<p>Link walking using REST API:</p>
<pre class="brush: fsharp">
let riakurl = "http://myriakhost1:8098"
let restClient = new HttpClient()
type LinkWalkSpec =
{ bucket: string; tag: string; keep: string; }
member x.Link = (sprintf "%s,%s,%s" x.bucket x.tag x.keep)
let linkWalker url bucket key (links:LinkWalkSpec list) =
let baseurl = sprintf "%s/buckets/%s/keys/%s" url bucket key
let rec buildLinkWalkUrl (linklist:LinkWalkSpec list) baseUrl =
match linklist with
| [] -> baseUrl
| [x] -> sprintf "%s/%s" baseUrl (x.Link)
| h::t -> let newUrl = sprintf "%s/%s" baseUrl (h.Link)
buildLinkWalkUrl t newUrl
buildLinkWalkUrl links baseurl
|> restClient.GetStringAsync
// Equiv to : curl http://localhost:8091/riak/cages/2/_,next_to,1/_,_,_
[{bucket="_";tag="next_to";keep="1"};{bucket="_";tag="_";keep="_"}]
|> linkWalker riakurl "cages" "2"
</pre>
<p>Link walking results from using REST API:</p>
<pre class="brush: plain">
val it : Task<string> =
System.Threading.Tasks.Task`1[System.String]
{AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 1;
IsCanceled = false;
IsCompleted = false;
IsFaulted = false;
Result = "
--CveXyss6PAqBxOOWxeWBCf6eXii
Content-Type: multipart/mixed; boundary=AvYXKrJYDlkeNxh1bQyqDvBAuBF
--AvYXKrJYDlkeNxh1bQyqDvBAuBF
X-Riak-Vclock: a85hYGBgzGDKBVIcqW/v8Qdy5rhnMCUy5rEy9Oi+P8WXBQA=
Location: /buckets/cages/keys/1
Content-Type: application/json
Link: </buckets/animals/keys/polly>; riaktag="contains", </buckets/cages>; rel="up"
Etag: 6gyXgkIgzvwBGRRotqHK3b
Last-Modified: Fri, 26 Apr 2013 16:55:40 GMT
{"room":101}
--AvYXKrJYDlkeNxh1bQyqDvBAuBF--
--CveXyss6PAqBxOOWxeWBCf6eXii
Content-Type: multipart/mixed; boundary=SaWqmmho48dzhMDmJy3BVcCWrzu
--SaWqmmho48dzhMDmJy3BVcCWrzu
X-Riak-Vclock: a85hYGBgzGDKBVIcqW/v8Qdy5rhnMCUy5rEyPDvYcIovCwA=
Location: /buckets/animals/keys/polly
Content-Type: application/json; charset=utf-8
Link: </buckets/animals>; rel="up"
Etag: 2BTveSKTYDNOZNCiOxyryw
Last-Modified: Fri, 01 Feb 2013 22:47:02 GMT
{ "nickname" : "Sweet Polly Purebred" , "breed" : "Purebred" }
--SaWqmmho48dzhMDmJy3BVcCWrzu--
--CveXyss6PAqBxOOWxeWBCf6eXii--
";
Status = RanToCompletion;}
</pre>
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com8tag:blogger.com,1999:blog-18281936.post-89529001991188713602013-04-26T06:00:00.000-07:002013-04-29T08:40:43.915-07:00Exploring Riak with F# and CorrugatedIron<p>Many thanks to <a href="http://www.blogger.com/profile/09971721667055138594">David</a> and <a href="http://buffered.io/">OJ</a> for recommending <a href="http://corrugatediron.org/">CorrugatedIron</a> .Net Riak client library. The following blog entry is to document my experiments with Riak CRUD operations using CorrugatedIron.</p>
<h3>Pinging RIAK</h3>
<p>I tested CorrugatedIron with F# script and here is the setup code along with testing ping capability:</p>
<pre class="brush: fsharp">
// Needed to load the following libraries to get F# script to work
#r @"c:\dev\FsRiak\packages\CorrugatedIron.1.3.0\lib\net40\CorrugatedIron.dll"
#r @"c:\dev\FsRiak\packages\protobuf-net.2.0.0.621\lib\net40\protobuf-net.dll"
#r @"c:\dev\FsRiak\packages\Newtonsoft.Json.4.5.11\lib\net40\Newtonsoft.Json.dll"
open CorrugatedIron
open CorrugatedIron.Models
open Newtonsoft.Json
open System
// Setup connections
let cluster = RiakCluster.FromConfig("riakConfig", @"c:\dev\FsRiak\App.config");
let client = cluster.CreateClient();
// Ping the Riak Cluster
client.Ping()
</pre>
<p>Here is my <code>App.config</code> file used by CorrugatedIron:</p>
<pre class="brush: xml">
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="riakConfig" type="CorrugatedIron.Config.RiakClusterConfiguration, CorrugatedIron" />
</configSections>
<riakConfig nodePollTime="5000" defaultRetryWaitTime="200" defaultRetryCount="3">
<nodes>
<node name="dev1" hostAddress="mydevhost-a" pbcPort="8087" restScheme="http" restPort="8098" poolSize="10" />
<node name="dev2" hostAddress="mydevhost-b" pbcPort="8087" restScheme="http" restPort="8098" poolSize="10" />
<node name="dev3" hostAddress="mydevhost-c" pbcPort="8087" restScheme="http" restPort="8098" poolSize="10" />
</nodes>
</riakConfig>
</configuration>
</pre>
<p>Here's the result from running ping:</p>
<pre class="brush: plain">
val it : RiakResult = CorrugatedIron.RiakResult {ErrorMessage = null;
IsSuccess = true;
ResultCode = Success;}
</pre>
<h3>Get List of Buckets</h3>
<p>The following method call gets you the list of buckets along with metadata for the call status:
<pre class="brush: fsharp">
client.ListBuckets()
</pre>
<p>This method returns the following <code>RiakResult</code> object:
<pre class="brush: plain">
val it : RiakResult<seq<string>> =
CorrugatedIron.RiakResult`1[System.Collections.Generic.IEnumerable`1[System.String]]
{ErrorMessage = null;
IsSuccess = true;
ResultCode = Success;
Value = seq ["photos"; "favs"; "animals"; "cages"; ...];}
</pre>
<h3>Get Bucket Keys</h3>
<p>Getting a list of keys for a bucket is also simple:</p>
<pre class="brush: fsharp">
client.ListKeys("animals")
</pre>
<p>For the novice, this library warns you to not to do this in production environments....</p>
<pre class="brush: plain">
*** [CI] -> ListKeys is an expensive operation and should not be used in Production scenarios. ***
val it : RiakResult<seq<string>> =
CorrugatedIron.RiakResult`1[System.Collections.Generic.IEnumerable`1[System.String]]
{ErrorMessage = null;
IsSuccess = true;
ResultCode = Success;
Value = seq ["ace"; "polly"];}
</pre>
<h3>Retrieve Content from Riak</h3>
<p>Getting a value from Riak is pretty easy with this library:
<pre class="brush: fsharp">
client.Get("animals","ace")
</pre>
<p>A dump of the return object shows the actual data plus metadata about the <code>Get</code> operation:
<pre class="brush: plain">
val it : RiakResult<Models.RiakObject> =
CorrugatedIron.RiakResult`1[CorrugatedIron.Models.RiakObject]
{ErrorMessage = null;
IsSuccess = true;
ResultCode = Success;
Value = CorrugatedIron.Models.RiakObject;}
</pre>
<p>A deeper dive into the <code>Value</code> field of <code>RiakResult</code> object gives the following:</p>
<pre class="brush: plain">
val it : Models.RiakObject =
CorrugatedIron.Models.RiakObject
{BinIndexes = dict [];
Bucket = "animals";
CharSet = null;
ContentEncoding = null;
ContentType = "application/json";
HasChanged = false;
IntIndexes = dict [];
Key = "ace";
LastModified = 1359744019u;
LastModifiedUsec = 788127u;
Links = seq [];
Siblings = seq [];
UserMetaData = dict [];
VTag = "7aPFusRQHlQ36ZP6G6GSyE";
VTags = seq ["7aPFusRQHlQ36ZP6G6GSyE"];
Value = [|123uy; 32uy; 34uy; 110uy; 105uy; 99uy; 107uy; 110uy; 97uy;
109uy; 101uy; 34uy; 32uy; 58uy; 32uy; 34uy; 84uy; 104uy; 101uy;
32uy; 87uy; 111uy; 110uy; 100uy; 101uy; 114uy; 32uy; 68uy;
111uy; 103uy; 34uy; 32uy; 44uy; 32uy; 34uy; 98uy; 114uy; 101uy;
101uy; 100uy; 34uy; 32uy; 58uy; 32uy; 34uy; 71uy; 101uy; 114uy;
109uy; 97uy; 110uy; 32uy; 83uy; 104uy; 101uy; 112uy; 104uy;
101uy; 114uy; 100uy; 34uy; 32uy; 125uy|];
VectorClock = [|107uy; 206uy; 97uy; 96uy; 96uy; 96uy; 204uy; 96uy; 202uy;
5uy; 82uy; 28uy; 172uy; 90uy; 225uy; 175uy; 2uy; 57uy;
59uy; 156uy; 51uy; 152uy; 18uy; 25uy; 243uy; 88uy; 25uy;
132uy; 59uy; 26uy; 78uy; 241uy; 101uy; 1uy; 0uy|];}
</pre>
<p>The data I really wanted is embedded in another <code>Value</code> field where it is represented as an array of bytes, which is not the final format I want. I wanted to get back the JSON representation of the data I put in. In my previous blog, I had rolled my own JSON serializer/deserializer, but now I wanted to leverage the other pieces of library bundle by CorrugateIron, namely <a href="http://james.newtonking.com/pages/json-net.aspx">Json.NET</a>. To do so,I define the <code>Animal</code> type and use Json.NET serializer/deserializer to convert between the objects and it's corresponding JSON representation</p>
<pre class="brush: fsharp">
type Animal =
{ nickname: string; breed: string}
// Getting ace from animals bucket
client.Get("animals","ace").Value.GetObject<Animal>()
</pre>
<h3>Adding Content to Riak</h3>
<p>Adding content to Riak is pretty easy also, after you define a specific type for Json.NET to serialize the fields of that type:
<pre class="brush: fsharp">
new RiakObject("animals","delta",{nickname="Snoopy"; breed="Beagle"})
|> client.Put
</pre>
<p>If you dump the RiakResult object and the Value field of RiakResult you get the following:</p>
<pre class="brush: plain">
val it : RiakResult<RiakObject> =
CorrugatedIron.RiakResult`1[CorrugatedIron.Models.RiakObject]
{ErrorMessage = null;
IsSuccess = true;
ResultCode = Success;
Value = CorrugatedIron.Models.RiakObject;}
val it : RiakObject =
CorrugatedIron.Models.RiakObject
{BinIndexes = dict [];
Bucket = "animals";
CharSet = null;
ContentEncoding = null;
ContentType = "application/json";
HasChanged = false;
IntIndexes = dict [];
Key = "delta";
LastModified = 1366930771u;
LastModifiedUsec = 271921u;
Links = seq [];
Siblings = seq [];
UserMetaData = dict [];
VTag = "OvZlH7bsYKdO8zL76QdDY";
VTags = seq ["OvZlH7bsYKdO8zL76QdDY"];
Value = [|123uy; 34uy; 110uy; 105uy; 99uy; 107uy; 110uy; 97uy; 109uy;
101uy; 34uy; 58uy; 34uy; 83uy; 110uy; 111uy; 111uy; 112uy;
121uy; 34uy; 44uy; 34uy; 98uy; 114uy; 101uy; 101uy; 100uy; 34uy;
58uy; 34uy; 66uy; 101uy; 97uy; 103uy; 108uy; 101uy; 34uy; 125uy|];
VectorClock = [|107uy; 206uy; 97uy; 96uy; 96uy; 96uy; 204uy; 96uy; 202uy;
5uy; 82uy; 28uy; 169uy; 111uy; 239uy; 241uy; 7uy; 114uy;
230uy; 184uy; 103uy; 48uy; 37uy; 50uy; 230uy; 177uy; 50uy;
4uy; 27uy; 190uy; 59uy; 197uy; 151uy; 5uy; 0uy|];}
</pre>
<p>To verify interoperability, I would check the newly added data with curl:</p>
<pre class="brush: plain">
$ curl -X GET http://192.168.56.1:8098/riak/animals/delta
$ {"nickname":"Snoopy","breed":"Beagle"}
</pre>
<h3>Delete Riak Contents</h3>
<p>Delete content by calling the intuitively named <code>Delete</code> method:</p>
<pre class="brush: fsharp">
client.Delete("animals","delta")
</pre>
<p>One thing that I wasn't sure is how CorrugatedIron talks to the Riak clusters. In my simple REST API example, I know exactly which host I'm talking to since I explicitly specified the url. For CorrugatedIron, I configure a pool of connections in the app.config file. I wasn't sure which of the nodes CorrugatedIron was talking to. I fire up <a href="http://www.wireshark.org/">Wireshark</a> and notice that I'm connected to the Riak cluster via port 8087, which is through the Protocol Buffers Client (PBC) interface...which explains the need to load the PBC libraries. This Protocol Buffers client is something new to me and the bundled <a href="http://code.google.com/p/protobuf-net/">protobuf-net</a> library seemed to be the code developed by Google (Many thanks to OJ for correcting me on this, this is NOT a Google library but a library written by <a href="marcgravell.blogspot.com">Marc Gravell</a>). This, in turn led me to look at Google <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a> . This little side jaunt into CorrugatedIron library led to the discovery (for me) of a whole new set of network communication protocols. In any case, after checking Wireshark output, it seems that the requests are spread out to the different nodes and not restricted to a single node. I'm guessing that there are some builtin load balancer code in CorrugatedIron that sends my request to different nodes in the Riak cluster.</p>
<p>For the basic CRUD operations on Riak, CorrugatedIron has made it easier to work with Riak then having to come up with my own helper functions. This has been a good start and I hope work through more of the Riak examples from the book <a href="http://www.amazon.com/exec/obidos/ASIN/1934356921/techie2biz-20">Seven Databases in Seven Weeks</a> with CorrugatedIron in future blog posts.</p>
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com2tag:blogger.com,1999:blog-18281936.post-78242969475782931042013-04-15T06:00:00.000-07:002013-04-15T06:00:00.820-07:00Exploring Riak with F#<p>I have embraced the
<a href="http://memeagora.blogspot.com/2006/12/polyglot-programming.html">
Polyglot Programming</a> for quite a while already. This year, I wanted to
tackle <a href="http://martinfowler.com/bliki/PolyglotPersistence.html">Polyglot Persistence</a>.
RDBMS has dominated my world view of persistence layer with everything else as second class
citizens. I thought it was time to expand my world view of persistence layers, especially
with the burgeoning popularity of NoSQL movement. To begin my exploration, I picked up the book
<a href="http://www.amazon.com/exec/obidos/ASIN/1934356921/techie2biz-20">
Seven Databases in Seven Weeks</a> by Eric Redmond and Jim Wilson and started with
the first NoSQL persistence layer in the book, which was <a href="http://basho.com/products/riak-overview/">Riak</a>.
According to the book:</p>
<blockquote>
Riak is a distributed key-value database where values can be anything-from plain text, JSON, or XML to
images or video clips-all accessible through a simple HTTP interface.
</blockquote>
<h3>Setting up Riak</h3>
<p>I deployed riak on 3 servers for testing purposes. In setting up the Riak clusters,
I ran into the following errors:</p>
<pre class="brush: plain">
10:42:57.339 [error] gen_server riak_core_capability terminated with reason: no function clause matching orddict:fetch('riak@192.168.56.1', [
{'riak@127.0.0.1',[{{riak_core,staged_joins},[true,false]},{{riak_core,vnode_routing},[proxy,legacy]},...]}]) line 72
/users/domains/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
</pre>
<p>A quick Google search brought up the following link
Googling the web, I got this link:
<a href="http://blog.alwayshere.info/2012/11/riak-error-genserver-riakcorecapability.html">
http://blog.alwayshere.info/2012/11/riak-error-genserver-riakcorecapability.html</a>
I originally started riak with 127.0.0.1 address. Then I made the modification
to the configuration as documented in <a href="http://docs.basho.com/riak/latest/cookbooks/Basic-Cluster-Setup/">
http://docs.basho.com/riak/latest/cookbooks/Basic-Cluster-Setup/</a> in trying to setup my 3 server Riak
into a cluster. To fix my error, I had to go to ./data/ring folder and delete everything in there,
then everything works as expected.</p>
<p>Most of the examples in the book leveraged curl. However, I learn best if I tried
to work the examples in another way. I tried the examples in Clojure, using ClojureWerkz's
<a href="http://clojureriak.info/">Welle</a>. I liked ClojureWerkz's Welle wrappers to Riak
and would probably use it if I had to develop on Java platform. I also wanted to work with
Riak from .NET platform and hence I'm using F# to explore Riak. The following examples where done on
Visual Studio 2010 with <a href="http://www.asp.net/mvc/mvc4">ASP.NET MVC 4</a> installed. This also gives me a chance to take the REST API in ASP.NET MVC4 for a spin.
</p>
<h3>Pinging RIAK</h3>
<p>The very first example in the book is to ping the Riak cluster, here's how
I implemented it in F#</p>
<pre class="brush: fsharp">
open System.Net.Http
open System.Threading.Tasks
// My 3 instances of Riak
let riak1_url = "http://192.168.56.1:8098"
let riak2_url = "http://192.168.56.2:8098"
let riak3_url = "http://192.168.56.3:8098"
// Pick one to work with
let riakurl = riak1_url
let client = new HttpClient()
let ping () = client.GetAsync(sprintf "%s/ping" riakurl)
ping()
</pre>
<p>Running the ping, I would get the following response from F#:</p>
<pre class="brush: plain">
val it : Task<HttpResponseMessage> =
System.Threading.Tasks.Task`1[System.Net.Http.HttpResponseMessage]
{AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 4;
IsCanceled = false;
IsCompleted = false;
IsFaulted = false;
Result = StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Vary: Accept-Encoding
Date: Wed, 30 Jan 2013 22:20:11 GMT
Server: MochiWeb/1.1
Server: WebMachine/1.9.0
Server: (someone had painted it blue)
Content-Length: 422
Content-Type: application/json
};
Status = RanToCompletion;}
</pre>
<h3>Adding Content to RIAK</h3>
<p>Let's start by putting some stuff into Riak with the following snippet of code</p>
<pre class="brush: fsharp">
let put bucket key content =
let put_url= sprintf "%s/riak/%s/%s" riakurl bucket key
client.PutAsync(put_url,content)
let put_html bucket key html =
let put_url= sprintf "%s/riak/%s/%s" riakurl bucket key
let content = new StringContent(html)
content.Headers.ContentType.MediaType <- "text/html"
put bucket key content
"<html><body><h1>My latest favorite DB is RIAK</h1></body></html>"
|> put_html "favs" "db"
</pre>
<p>Running the above script gets the following response:</p>
<pre class="brush: plain">
val it : Task<HttpResponseMessage> =
System.Threading.Tasks.Task`1[System.Net.Http.HttpResponseMessage]
{AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 5;
IsCanceled = false;
IsCompleted = false;
IsFaulted = false;
Result = StatusCode: 204, ReasonPhrase: 'No Content', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Vary: Accept-Encoding
Date: Fri, 01 Feb 2013 18:07:24 GMT
Server: MochiWeb/1.1
Server: WebMachine/1.9.0
Server: (someone had painted it blue)
Content-Length: 0
Content-Type: text/html; charset=utf-8
};
Status = RanToCompletion;}
</pre>
<p>We got the 204 code as explained in the book.
To test that Riak has stored this new content,
simply point to any of the Riak instances, (e.g.
<code>http://192.168.56.3:8098/riak/favs/db</code>)
with a browser and you should see the webpage that was put into the first Riak server.
</p>
<p>
Here's the sample code to put JSON data into Riak:
</p>
<pre class="brush: fsharp">
// Simple utility to generate JSON - should really use a real JSON library
let tojson data =
data |> Seq.map (fun (k,v) -> sprintf "\"%s\" : \"%s\"" k v)
|> Seq.reduce (sprintf "%s , %s")
|> sprintf "{ %s }"
let put_json bucket key jsondata =
let content = new StringContent(tojson jsondata)
content.Headers.ContentType.MediaType <- "application/json"
put bucket key content
[("nickname","The Wonder Dog"); ("breed","German Shepherd")]
|> put_json "animals" "ace"
</pre>
<p>Again, you can check that it's stored in Riak by pointing the browser to:
<code>http://192.168.56.3:8098/riak/animals/ace</code> and you should get back:
<pre class="brush: plain">
{ "nickname" : "The Wonder Dog" , "breed" : "German Shepherd" }
</pre>
<h3>Removing Content from RIAK</h3>
<p>Here's a snippet of script to remove a content from Riak</p>
<pre class="brush: fsharp">
let delete bucket key =
let delete_url= sprintf "%s/riak/%s/%s" riakurl bucket key
client.DeleteAsync(delete_url)
delete "animals" "ace"
</pre>
<h3>Getting Bucket Keys</h3>
<p>To get all keys in a bucket</p>
<pre class="brush: fsharp">
let get_keys bucket =
let get_url = sprintf "%s/riak/%s?keys=true" riakurl bucket
get_url |> client.GetStringAsync
let results = get_keys "animals"
printfn "%s" results.Result
</pre>
<p>The above script would return the following (reformatted for legibility purposes):</p>
<pre class="brush: plain">
{"props":{"name":"animals",
"allow_mult":false,
"basic_quorum":false,
"big_vclock":50,
"chash_keyfun":{"mod":"riak_core_util",
"fun":"chash_std_keyfun"},
"dw":"quorum",
"last_write_wins":false,
"linkfun":{"mod":"riak_kv_wm_link_walker",
"fun":"mapreduce_linkfun"},
"n_val":3,
"notfound_ok":true,
"old_vclock":86400,
"postcommit":[],
"pr":0,
"precommit":[],
"pw":0,
"r":"quorum",
"rw":"quorum",
"small_vclock":50,
"w":"quorum",
"young_vclock":20},
"keys":["ace","polly"]}
</pre>
<h3>Retrieving Content from Riak</h3>
<p>To retrieve content:</p>
<pre class="brush: fsharp">
let get bucket key =
let get_url = sprintf "%s/riak/%s/%s/" riakurl bucket key
get_url |> client.GetStringAsync
let results = get "animals" "ace"
printfn "%s" results.Result
</pre>
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com3tag:blogger.com,1999:blog-18281936.post-89624669620930016702013-03-25T06:00:00.000-07:002013-03-25T06:00:11.523-07:00Compare Tibco EMS Performance Under Load with Clojure<style>
#mytable {
width: 700px;
padding: 0;
margin: 0;
}
caption {
padding: 0 0 5px 0;
width: 700px;
font: italic 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
text-align: right;
}
th {
font: bold 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #4f6b72;
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
border-top: 1px solid #C1DAD7;
letter-spacing: 2px;
text-transform: uppercase;
text-align: left;
padding: 6px 6px 6px 12px;
background: #CAE8EA url(images/bg_header.jpg) no-repeat;
}
th.nobg {
border-top: 0;
border-left: 0;
border-right: 1px solid #C1DAD7;
background: none;
}
td {
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
background: #fff;
padding: 6px 6px 6px 12px;
color: #4f6b72;
text-align:center;
}
td.alt {
background: #F5FAFA;
color: #797268;
text-align:center;
}
th.spec {
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #fff url(images/bullet1.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
th.specalt {http://www.blogger.com/post-edit.g?blogID=18281936&postID=4559158457660466923
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #f5fafa url(images/bullet2.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #797268;
}
</style>
<p>We have migrated our Tibco EMS infrastructure from the older Sun V490 servers to the newer HP ProLiant DL380 G7 servers.
One of the concerns was how does the new servers perform under load compared with the older servers. I needed to create a load
test that compared the performance of the two different types of servers. I turned to Clojure to help me put together this load test
and here is the load test code:
<hr/>
<h3>Load Test Script in Clojure</h3>
<pre class="brush: clojure;">
; Leveraging Apache Commons to generate the random message
(def msg64 (RandomStringUtils/randomAlphanumeric 64))
(def msg128 (RandomStringUtils/randomAlphanumeric 128))
(def msg256 (RandomStringUtils/randomAlphanumeric 256))
(def msg64k (RandomStringUtils/randomAlphanumeric (* 64 1024)))
(def msg128k (RandomStringUtils/randomAlphanumeric (* 128 1024)))
(def msg256k (RandomStringUtils/randomAlphanumeric (* 256 1024)))
(def msg512k (RandomStringUtils/randomAlphanumeric (* 512 1024)))
(def msg1mb (RandomStringUtils/randomAlphanumeric (* 1024 1024)))
(def persistent-delivery (.intValue (javax.jms.DeliveryMode/PERSISTENT)))
; Send messages
(defn send-messages [server-url user password queue-name data iterations]
(with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
(.createQueueConnection user password))]
(let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE)
queue (.createQueue session queue-name)]
(with-open [sender (.createSender session queue)]
(dotimes [_ iterations]
(let [message (.createTextMessage session)]
(.setJMSDeliveryMode message persistent-delivery)
(.setText message data)
(.send sender message)))))))
(def test-scenarios (list
{:label "64 bytes" :msg msg64 :n 5000 }
{:label "128 bytes" :msg msg128 :n 5000 }
{:label "256 bytes" :msg msg256 :n 5000 }
{:label "64KB bytes" :msg msg64k :n 2000 }
{:label "128KB bytes" :msg msg128k :n 1000 }
{:label "256KB bytes" :msg msg256k :n 1000 }
{:label "512KB bytes" :msg msg512k :n 1000 }
{:label "1MB bytes" :msg msg1mb :n 1000 }))
(doseq [my-test test-scenarios]
(let [label (:label my-test)
message (:msg my-test)
n (:n my-test)]
(println "\n\n")
(println (str "Testing for " label " sized messages with n = " n))
(time (send-messages server-url username password test-queue message n))))
</pre>
<p>Here's the message consumer part of the load test:</p>
<hr/>
<h3>Message Consumer in Clojure</h3>
<pre class="brush: clojure;">
; Consume Queue Text messages asynchronously
(defn get-queue-text-messages [server-url user password queue-name process-message]
(future
(with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
(.createQueueConnection user password))]
(let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE)
queue (.createQueue session queue-name)]
(with-open [receiver (.createReceiver session queue)]
(.start connection)
(loop [acc 0]
(process-message (.receive receiver) acc)
(recur (+ acc 1))))))))
; Dump just the message id
(defn dump-message-id [message n]
(println (str n)))
; Create function aliases with connection information embedded
(defn consume-messages [queue-name message-processor]
(get-queue-text-messages server-url username password queue-name message-processor))
; Start consuming messages asynchronously
(consume-messages queueName dump-message-id)
</pre>
<p>Here are the test results:</p>
<hr/>
<h3>Load Test Result on Sun V490</h3>
<table><thead><th>Message Size</th><th>Number of Messages</th><th>Process Time (ms)</th><th>Throughput (msg/s)</th><th>Throughput (MB/s)</th></thead>
<tr>
<td>64</td>
<td>5,000</td>
<td>2,772.84</td>
<td>1,803.21</td>
<td>0.11</td>
</tr>
<tr>
<td>128</td>
<td>5,000</td>
<td>1,943.23</td>
<td>2,573.04</td>
<td>0.31</td>
</tr>
<tr>
<td>256</td>
<td>5,000</td>
<td>1,899.24</td>
<td>2,632.63</td>
<td>0.64</td>
</tr>
<tr>
<td>64K</td>
<td>1,000</td>
<td>3,020.98</td>
<td>331.02</td>
<td>20.69</td>
</tr>
<tr>
<td>128K</td>
<td>1,000</td>
<td>4,414.00</td>
<td>226.55</td>
<td>28.32</td>
</tr>
<tr>
<td>256K</td>
<td>1,000</td>
<td>20,911.61</td>
<td>47.82</td>
<td>11.96</td>
</tr>
<tr>
<td>512K</td>
<td>1,000</td>
<td>39,213.23</td>
<td>25.50</td>
<td>12.75</td>
</tr>
<tr>
<td>1MB</td>
<td>1,000</td>
<td>80,337.55</td>
<td>12.45</td>
<td>12.45</td>
</tr>
</table>
<hr/>
<h3>Load Test Result on HP ProLiant DL380 G7</h3>
<table><thead><th>Message Size</th><th>Number of Messages</th><th>Process Time (ms)</th><th>Throughput (msg/s)</th><th>Throughput (MB/s)</th></thead>
<tr>
<td>64</td>
<td>5,000</td>
<td>1,340.22</td>
<td>3,730.74</td>
<td>0.23</td>
</tr>
<tr>
<td>128</td>
<td>5,000</td>
<td>1,345.18</td>
<td>3,716.99</td>
<td>0.45</td>
</tr>
<tr>
<td>256</td>
<td>5,000</td>
<td>1,040.59</td>
<td>4,804.95</td>
<td>1.17</td>
</tr>
<tr>
<td>64K</td>
<td>1,000</td>
<td>2,178.69</td>
<td>458.99</td>
<td>28.69</td>
</tr>
<tr>
<td>128K</td>
<td>1,000</td>
<td>3,553.02</td>
<td>281.45</td>
<td>35.18</td>
</tr>
<tr>
<td>256K</td>
<td>1,000</td>
<td>7,490.54</td>
<td>133.50</td>
<td>33.38</td>
</tr>
<tr>
<td>512K</td>
<td>1,000</td>
<td>35,078.26</td>
<td>28.51</td>
<td>14.25</td>
</tr>
<tr>
<td>1MB</td>
<td>1,000</td>
<td>77,076.51</td>
<td>12.97</td>
<td>12.97</td>
</tr>
</table>
<p>From this test result, we can conclude that HP ProLiant DL380 G7 performed better than Sun V490 under load.</p>
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com10tag:blogger.com,1999:blog-18281936.post-25386255318430128232013-03-04T06:00:00.000-08:002013-03-04T06:00:14.914-08:00Retrieving VMware vSphere Performance Metrics Catalog with F#<p>vSphere collects a lot of performance metrics data on a VMware cluster.
Our problem was that we cannot find a complete documentation on what are all those
performance metrics data collected and what are the units of the collected metric data.
We have asked VMware professional services engineer about this and his response
was that VMware does not have the metrics data documented.</p>
<p>However, in Chapter 16 of the <a href="http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsdk_prog_guide.pdf">
vSphere Web Services Programming Guide</a>, it talked about the fact
that you can retrieve the metrics catalog from vSphere itself.
Although this document talks about retrieving the metrics catalog via vSphere
Web Services SDK, you can also retrieve the metrics catalog with PowerCLI.
Below is a F# script that retrieves vSphere metrics catalog:</p>
<pre class="brush: fsharp;">
#r @"C:\pkg\VMware\Infrastructure\vSphere PowerCLI\VMware.Vim.dll"
open System
open VMware.Vim
open System.Collections.Specialized
let client = new VimClient()
let service = client.Connect(serviceUrl)
// Must login to do anything - if you are getting null values, it means the session automatically timed out
client.Login(userId,password)
let printMetricCatalog () =
// counterId max is arbitrarily set, I tried other values
// and still produced the same number of returned metrics
let counterIds = seq {1 .. 1000}
let perfMgr = new PerformanceManager(client,client.ServiceContent.PerfManager)
let metrics = perfMgr.QueryPerfCounter(counterIds |> Seq.toArray)
let dumpAllMetricData (metric:PerfCounterInfo) =
printfn "-------------------------------------------------------------------\n"
printfn "Key : %i" metric.Key
printfn "Level : %A" metric.Level
printfn "PerDeviceLevel : %A" metric.PerDeviceLevel
printfn "RollupType : %A" metric.RollupType
printfn "StatsType : %A" metric.StatsType
printfn "Group.Label : %s" metric.GroupInfo.Label
printfn "Group.Key : %s" metric.GroupInfo.Key
printfn "Group.Summary : %s" metric.GroupInfo.Summary
printfn "NameInfo.Label : %s" metric.NameInfo.Label
printfn "NameInfo.Key : %s" metric.NameInfo.Key
printfn "NameInfo.Summary: %s" metric.NameInfo.Summary
printfn "UnitInfo.Label : %s" metric.UnitInfo.Label
printfn "UnitInfo.Key : %s" metric.UnitInfo.Key
printfn "UnitInfo.Summary: %s" metric.UnitInfo.Summary
let dumpMetricData (metric:PerfCounterInfo) = printfn "\t%s (%s) - %A stat, rollup: %A"
(metric.NameInfo.Label)
(metric.UnitInfo.Label)
(metric.StatsType)
(metric.RollupType)
// Break metrics catalog by metric groups
let metricGroups = metrics |> Seq.map (fun metric -> metric.GroupInfo.Label) |> Set.ofSeq |> Set.toSeq
// Dump short version of metrics catalog
metricGroups
|> Seq.iter (fun group -> printfn "\n--------------------------------------------------"
printfn " Metric Group : %s" group
printfn "--------------------------------------------------"
metrics
|> Seq.filter (fun metric -> metric.GroupInfo.Label = group)
|> Seq.iter dumpMetricData)
// Dump complete version of metrics catalog
metrics |> Seq.iter dumpAllMetricData
printMetricCatalog()
</pre>
<p>
Here's a dump of the metrics catalog reformatted to display as HTML tables:
</p>
<style>
#mytable {
width: 700px;
padding: 0;
margin: 0;
}
caption {
padding: 0 0 5px 0;
width: 700px;
font: italic 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
text-align: right;
}
th {
font: bold 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #4f6b72;
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
border-top: 1px solid #C1DAD7;
letter-spacing: 2px;
text-transform: uppercase;
text-align: left;
padding: 6px 6px 6px 12px;
background: #CAE8EA url(images/bg_header.jpg) no-repeat;
}
th.nobg {
border-top: 0;
border-left: 0;
border-right: 1px solid #C1DAD7;
background: none;
}
td {
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
background: #fff;
padding: 6px 6px 6px 12px;
color: #4f6b72;
}
td.alt {
background: #F5FAFA;
color: #797268;
}
th.spec {
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #fff url(images/bullet1.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
th.specalt {http://www.blogger.com/post-edit.g?blogID=18281936&postID=4559158457660466923
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #f5fafa url(images/bullet2.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #797268;
}
</style>
<h2>VMware vSphere Metrics Catalog</h2>
<hr/>
<h3>CPU</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>CPU Capacity Contention</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>Percent of time the VMs on this host are unable to run because they are contending for access to the physical CPU(s)</td></tr>
<tr><td>CPU Capacity Demand</td>
<td>MHz</td>
<td>absolute</td>
<td>average</td>
<td>The amount of CPU resources VMs on this host would use if there were no CPU contention or CPU limit</td></tr>
<tr><td>CPU Capacity Entitlement</td>
<td>MHz</td>
<td>absolute</td>
<td>average</td>
<td>CPU resources devoted by the ESX scheduler to virtual machines and resource pools</td></tr>
<tr><td>CPU Capacity Provisioned</td>
<td>MHz</td>
<td>absolute</td>
<td>average</td>
<td>Capacity in MHz of the physical CPU cores</td></tr>
<tr><td>CPU Capacity Usage</td>
<td>MHz</td>
<td>rate</td>
<td>average</td>
<td>CPU usage in MHz during the interval</td></tr>
<tr><td>CPU Core Count Contention</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>Time the VM is ready to run, but is unable to run due to co-scheduling constraints</td></tr>
<tr><td>CPU Core Count Provisioned</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>The number of physical cores provisioned to the entity</td></tr>
<tr><td>CPU Core Count Usage</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>The number of virtual processors running on the host</td></tr>
<tr><td>Co-stop</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>Time the VM is ready to run, but is unable to due to co-scheduling constraints</td></tr>
<tr><td>Core Utilization</td>
<td>Percent</td>
<td>rate</td>
<td>none</td>
<td>CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)</td></tr>
<tr><td>Core Utilization</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)</td></tr>
<tr><td>Core Utilization</td>
<td>Percent</td>
<td>rate</td>
<td>maximum</td>
<td>CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)</td></tr>
<tr><td>Core Utilization</td>
<td>Percent</td>
<td>rate</td>
<td>minimum</td>
<td>CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)</td></tr>
<tr><td>Demand</td>
<td>MHz</td>
<td>absolute</td>
<td>average</td>
<td>The amount of CPU resources a VM would use if there were no CPU contention or CPU limit</td></tr>
<tr><td>Entitlement</td>
<td>MHz</td>
<td>absolute</td>
<td>latest</td>
<td>CPU resources devoted by the ESX scheduler</td></tr>
<tr><td>Idle</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>Total time that the CPU spent in an idle state</td></tr>
<tr><td>Latency</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>Percent of time the VM is unable to run because it is contending for access to the physical CPU(s)</td></tr>
<tr><td>Max limited</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>Time the VM is ready to run, but is not run due to maxing out its CPU limit setting</td></tr>
<tr><td>Overlap</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>Time the VM was interrupted to perform system services on behalf of that VM or other VMs</td></tr>
<tr><td>Ready</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>Percentage of time that the virtual machine was ready, but could not get scheduled to run on the physical CPU</td></tr>
<tr><td>Reserved capacity</td>
<td>MHz</td>
<td>absolute</td>
<td>average</td>
<td>Total CPU capacity reserved by virtual machines</td></tr>
<tr><td>Run</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>Time the VM is scheduled to run</td></tr>
<tr><td>Swap wait</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>CPU time spent waiting for swap-in</td></tr>
<tr><td>System</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>Amount of time spent on system processes on each virtual CPU in the virtual machine</td></tr>
<tr><td>Total</td>
<td>MHz</td>
<td>rate</td>
<td>average</td>
<td>Total amount of CPU resources of all hosts in the cluster</td></tr>
<tr><td>Total capacity</td>
<td>MHz</td>
<td>absolute</td>
<td>average</td>
<td>Total CPU capacity reserved by and available for virtual machines</td></tr>
<tr><td>Usage</td>
<td>Percent</td>
<td>rate</td>
<td>none</td>
<td>CPU usage as a percentage during the interval</td></tr>
<tr><td>Usage</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>CPU usage as a percentage during the interval</td></tr>
<tr><td>Usage</td>
<td>Percent</td>
<td>rate</td>
<td>minimum</td>
<td>CPU usage as a percentage during the interval</td></tr>
<tr><td>Usage</td>
<td>Percent</td>
<td>rate</td>
<td>maximum</td>
<td>CPU usage as a percentage during the interval</td></tr>
<tr><td>Usage in MHz</td>
<td>MHz</td>
<td>rate</td>
<td>none</td>
<td>CPU usage in megahertz during the interval</td></tr>
<tr><td>Usage in MHz</td>
<td>MHz</td>
<td>rate</td>
<td>average</td>
<td>CPU usage in megahertz during the interval</td></tr>
<tr><td>Usage in MHz</td>
<td>MHz</td>
<td>rate</td>
<td>minimum</td>
<td>CPU usage in megahertz during the interval</td></tr>
<tr><td>Usage in MHz</td>
<td>MHz</td>
<td>rate</td>
<td>maximum</td>
<td>CPU usage in megahertz during the interval</td></tr>
<tr><td>Used</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>Total CPU usage</td></tr>
<tr><td>Utilization</td>
<td>Percent</td>
<td>rate</td>
<td>none</td>
<td>CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)</td></tr>
<tr><td>Utilization</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)</td></tr>
<tr><td>Utilization</td>
<td>Percent</td>
<td>rate</td>
<td>maximum</td>
<td>CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)</td></tr>
<tr><td>Utilization</td>
<td>Percent</td>
<td>rate</td>
<td>minimum</td>
<td>CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)</td></tr>
<tr><td>Wait</td>
<td>Millisecond</td>
<td>delta</td>
<td>summation</td>
<td>Total CPU time spent in wait state</td></tr>
<tr><td>Worst case allocation</td>
<td>MHz</td>
<td>absolute</td>
<td>latest</td>
<td>Amount of CPU resources allocated to the virtual machine or resource pool based on the total cluster capacity and the resource configuration of the resource hierarchy</td></tr>
</table>
<hr/>
<h3>Cluster services</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>CPU fairness</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Fairness of distributed CPU resource allocation</td></tr>
<tr><td>Current failover level</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>vSphere HA number of failures that can be tolerated</td></tr>
<tr><td>Effective CPU resources</td>
<td>MHz</td>
<td>rate</td>
<td>average</td>
<td>Total available CPU resources of all hosts within a cluster</td></tr>
<tr><td>Effective memory resources</td>
<td>MB</td>
<td>absolute</td>
<td>average</td>
<td>Total amount of machine memory of all hosts in the cluster that is available for use for virtual machine memory and overhead memory</td></tr>
<tr><td>Memory fairness</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Aggregate available memory resources of all the hosts within a cluster</td></tr>
</table>
<hr/>
<h3>Datastore</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Average read requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of read commands issued per second to the datastore during the collection interval</td></tr>
<tr><td>Average write requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of write commands issued per second to the datastore during the collection interval</td></tr>
<tr><td>Highest latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>latest</td>
<td>Highest latency value across all datastores used by the host</td></tr>
<tr><td>Read latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>The average time a read from the datastore takes</td></tr>
<tr><td>Read rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of reading data from the datastore</td></tr>
<tr><td>Storage DRS datastore bytes read</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore bytes read</td></tr>
<tr><td>Storage DRS datastore bytes written</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore bytes written</td></tr>
<tr><td>Storage DRS datastore normalized read latency</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore normalized read latency</td></tr>
<tr><td>Storage DRS datastore normalized write latency</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore normalized write latency</td></tr>
<tr><td>Storage DRS datastore outstanding read requests</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore outstanding read requests</td></tr>
<tr><td>Storage DRS datastore outstanding write requests</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore outstanding write requests</td></tr>
<tr><td>Storage DRS datastore read I/O rate</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore read I/O rate</td></tr>
<tr><td>Storage DRS datastore read workload metric</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore metric for read workload model</td></tr>
<tr><td>Storage DRS datastore write I/O rate</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore write I/O rate</td></tr>
<tr><td>Storage DRS datastore write workload metric</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS datastore metric for write workload model</td></tr>
<tr><td>Storage I/O Control aggregated IOPS</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>Storage I/O Control aggregated IOPS</td></tr>
<tr><td>Storage I/O Control datastore maximum queue depth</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage I/O Control datastore maximum queue depth</td></tr>
<tr><td>Storage I/O Control normalized latency</td>
<td>Microsecond</td>
<td>absolute</td>
<td>average</td>
<td>Storage I/O Control size-normalized I/O latency</td></tr>
<tr><td>Write latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>The average time a write to the datastore takes</td></tr>
<tr><td>Write rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of writing data to the datastore</td></tr>
<tr><td>busResets</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>busResets</td></tr>
<tr><td>commandsAborted</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>commandsAborted</td></tr>
<tr><td>contention</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>contention</td></tr>
<tr><td>usage</td>
<td>KBps</td>
<td>absolute</td>
<td>average</td>
<td>usage</td></tr>
</table>
<hr/>
<h3>Disk</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Average commands issued per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of SCSI commands issued per second during the collection interval</td></tr>
<tr><td>Average read requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of disk reads per second during the collection interval</td></tr>
<tr><td>Average write requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of disk writes per second during the collection interval</td></tr>
<tr><td>Bus resets</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of SCSI-bus reset commands issued during the collection interval</td></tr>
<tr><td>Capacity</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Configured size of the datastore</td></tr>
<tr><td>Command latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time taken during the collection interval to process a SCSI command issued by the Guest OS to the virtual machine</td></tr>
<tr><td>Commands issued</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of SCSI commands issued during the collection interval</td></tr>
<tr><td>Commands terminated</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of SCSI commands terminated during the collection interval</td></tr>
<tr><td>Disk SCSI Reservation Conflicts</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of SCSI reservation conflicts for the LUN during the collection interval</td></tr>
<tr><td>Disk SCSI Reservation Conflicts %</td>
<td>Percent</td>
<td>absolute</td>
<td>average</td>
<td>Number of SCSI reservation conflicts for the LUN as a percent of total commands during the collection interval</td></tr>
<tr><td>Disk Throughput Contention</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time for an I/O operation to complete</td></tr>
<tr><td>Disk Throughput Usage</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Aggregated disk I/O rate</td></tr>
<tr><td>Highest latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>latest</td>
<td>Highest latency value across all disks used by the host</td></tr>
<tr><td>Kernel command latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time, in milliseconds, spent by VMkernel to process each SCSI command</td></tr>
<tr><td>Kernel read latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time, in milliseconds, spent by VMKernel to process each SCSI read command</td></tr>
<tr><td>Kernel write latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time, in milliseconds, spent by VMKernel to process each SCSI write command</td></tr>
<tr><td>Maximum queue depth</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>Maximum queue depth</td></tr>
<tr><td>Overhead due to delta disk backings</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Storage overhead of a virtual machine or a datastore due to delta disk backings</td></tr>
<tr><td>Physical device command latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time, in milliseconds, to complete a SCSI command from the physical device</td></tr>
<tr><td>Physical device read latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time, in milliseconds, to read from the physical device</td></tr>
<tr><td>Physical device write latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time, in milliseconds, to write to the physical device</td></tr>
<tr><td>Queue command latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval</td></tr>
<tr><td>Queue read latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time spent in the VMkernel queue, per SCSI read command, during the collection interval</td></tr>
<tr><td>Queue write latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount time spent in the VMkernel queue, per SCSI write command, during the collection interval</td></tr>
<tr><td>Read latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time taken during the collection interval to process a SCSI read command issued from the Guest OS to the virtual machine</td></tr>
<tr><td>Read rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average number of kilobytes read from the disk each second during the collection interval</td></tr>
<tr><td>Read requests</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of disk reads during the collection interval</td></tr>
<tr><td>Space actually used</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Amount of space actually used by the virtual machine or the datastore</td></tr>
<tr><td>Space not shared</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Amount of space associated exclusively with a virtual machine</td></tr>
<tr><td>Space potentially used</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Amount of storage set aside for use by a datastore or a virtual machine</td></tr>
<tr><td>Usage</td>
<td>KBps</td>
<td>rate</td>
<td>none</td>
<td>Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.</td></tr>
<tr><td>Usage</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.</td></tr>
<tr><td>Usage</td>
<td>KBps</td>
<td>rate</td>
<td>minimum</td>
<td>Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.</td></tr>
<tr><td>Usage</td>
<td>KBps</td>
<td>rate</td>
<td>maximum</td>
<td>Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.</td></tr>
<tr><td>Write latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time taken during the collection interval to process a SCSI write command issued by the Guest OS to the virtual machine</td></tr>
<tr><td>Write rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average number of kilobytes written to disk each second during the collection interval</td></tr>
<tr><td>Write requests</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of disk writes during the collection interval</td></tr>
<tr><td>contention</td>
<td>Percent</td>
<td>absolute</td>
<td>average</td>
<td>contention</td></tr>
<tr><td>provisioned</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>provisioned</td></tr>
<tr><td>usage</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>usage</td></tr>
</table>
<hr/>
<h3>Management agent</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>CPU usage</td>
<td>MHz</td>
<td>rate</td>
<td>average</td>
<td>Amount of Service Console CPU usage</td></tr>
<tr><td>Memory swap in</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Amount of memory that is swapped in for the Service Console</td></tr>
<tr><td>Memory swap out</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Amount of memory that is swapped out for the Service Console</td></tr>
<tr><td>Memory swap used</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Sum of the memory swapped by all powered-on virtual machines on the host</td></tr>
<tr><td>Memory used</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of total configured memory that is available for use</td></tr>
</table>
<hr/>
<h3>Memory</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Active</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages</td></tr>
<tr><td>Active</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages</td></tr>
<tr><td>Active</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages</td></tr>
<tr><td>Active</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages</td></tr>
<tr><td>Active write</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory that is actively being written to by the VM</td></tr>
<tr><td>Balloon</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools</td></tr>
<tr><td>Balloon</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools</td></tr>
<tr><td>Balloon</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools</td></tr>
<tr><td>Balloon</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools</td></tr>
<tr><td>Balloon target</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Target value set by VMkernal for the virtual machine's memory balloon size</td></tr>
<tr><td>Balloon target</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Target value set by VMkernal for the virtual machine's memory balloon size</td></tr>
<tr><td>Balloon target</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Target value set by VMkernal for the virtual machine's memory balloon size</td></tr>
<tr><td>Balloon target</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Target value set by VMkernal for the virtual machine's memory balloon size</td></tr>
<tr><td>Compressed</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory compressed by ESX</td></tr>
<tr><td>Compression rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of memory compression for the VM</td></tr>
<tr><td>Consumed</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of memory consumed by a virtual machine, host, or cluster</td></tr>
<tr><td>Consumed</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory consumed by a virtual machine, host, or cluster</td></tr>
<tr><td>Consumed</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of memory consumed by a virtual machine, host, or cluster</td></tr>
<tr><td>Consumed</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of memory consumed by a virtual machine, host, or cluster</td></tr>
<tr><td>Decompression rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of memory decompression for the VM</td></tr>
<tr><td>Entitlement</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler</td></tr>
<tr><td>Granted</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of machine memory or physical memory that is mapped for a virtual machine or a host</td></tr>
<tr><td>Granted</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of machine memory or physical memory that is mapped for a virtual machine or a host</td></tr>
<tr><td>Granted</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of machine memory or physical memory that is mapped for a virtual machine or a host</td></tr>
<tr><td>Granted</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of machine memory or physical memory that is mapped for a virtual machine or a host</td></tr>
<tr><td>Heap</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>VMkernel virtual address space dedicated to VMkernel main heap and related data</td></tr>
<tr><td>Heap</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>VMkernel virtual address space dedicated to VMkernel main heap and related data</td></tr>
<tr><td>Heap</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>VMkernel virtual address space dedicated to VMkernel main heap and related data</td></tr>
<tr><td>Heap</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>VMkernel virtual address space dedicated to VMkernel main heap and related data</td></tr>
<tr><td>Heap free</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Free address space in the VMkernel's main heap</td></tr>
<tr><td>Heap free</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Free address space in the VMkernel's main heap</td></tr>
<tr><td>Heap free</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Free address space in the VMkernel's main heap</td></tr>
<tr><td>Heap free</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Free address space in the VMkernel's main heap</td></tr>
<tr><td>Host cache used for swapping</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Space used for caching swapped pages in the host cache</td></tr>
<tr><td>Host cache used for swapping</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Space used for caching swapped pages in the host cache</td></tr>
<tr><td>Host cache used for swapping</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Space used for caching swapped pages in the host cache</td></tr>
<tr><td>Host cache used for swapping</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Space used for caching swapped pages in the host cache</td></tr>
<tr><td>Latency</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>Percentage of time the VM is waiting to access swapped or compressed memory</td></tr>
<tr><td>Low free threshold</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Threshold of free host physical memory below which ESX will begin reclaiming memory from VMs through ballooning and swapping</td></tr>
<tr><td>Memory Capacity Contention</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>Percentage of time the VM is waiting to access swapped, compressed, or ballooned memory</td></tr>
<tr><td>Memory Capacity Entitlement</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler</td></tr>
<tr><td>Memory Capacity Provisioned</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Total amount of memory configured for the VM</td></tr>
<tr><td>Memory Capacity Usable</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of physical memory available for use by virtual machines on this host</td></tr>
<tr><td>Memory Capacity Usage</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of physical memory actively used</td></tr>
<tr><td>Memory Consumed by VMs</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of physical memory consumed by VMs on this host</td></tr>
<tr><td>Memory Consumed by userworlds</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of physical memory consumed by userworlds on this host</td></tr>
<tr><td>Memory Reserved Capacity %</td>
<td>Percent</td>
<td>absolute</td>
<td>average</td>
<td>Percent of memory that has been reserved either through VMkernel use, by userworlds, or due to VM memory reservations</td></tr>
<tr><td>Memory saved by zipping</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory (KB) saved due to memory zipping</td></tr>
<tr><td>Overhead</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Memory (KB) consumed by the virtualization infrastructure for running the VM</td></tr>
<tr><td>Overhead</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Memory (KB) consumed by the virtualization infrastructure for running the VM</td></tr>
<tr><td>Overhead</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Memory (KB) consumed by the virtualization infrastructure for running the VM</td></tr>
<tr><td>Overhead</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Memory (KB) consumed by the virtualization infrastructure for running the VM</td></tr>
<tr><td>Overhead touched</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Actively touched overhead memory (KB) reserved for use as the virtualization overhead for the VM</td></tr>
<tr><td>Reserved capacity</td>
<td>MB</td>
<td>absolute</td>
<td>average</td>
<td>Total amount of memory reservation used by powered-on virtual machines and vSphere services on the host</td></tr>
<tr><td>Reserved overhead</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Memory (KB) reserved for use as the virtualization overhead for the VM</td></tr>
<tr><td>Shared</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host</td></tr>
<tr><td>Shared</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host</td></tr>
<tr><td>Shared</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host</td></tr>
<tr><td>Shared</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host</td></tr>
<tr><td>Shared common</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host</td></tr>
<tr><td>Shared common</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host</td></tr>
<tr><td>Shared common</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host</td></tr>
<tr><td>Shared common</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host</td></tr>
<tr><td>State</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>One of four threshold levels representing the percentage of free memory on the host. The counter value determines swapping and ballooning behavior for memory reclamation.</td></tr>
<tr><td>Swap in</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount swapped-in to memory from disk</td></tr>
<tr><td>Swap in</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount swapped-in to memory from disk</td></tr>
<tr><td>Swap in</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount swapped-in to memory from disk</td></tr>
<tr><td>Swap in</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount swapped-in to memory from disk</td></tr>
<tr><td>Swap in from host cache</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of memory swapped-in from host cache</td></tr>
<tr><td>Swap in from host cache</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory swapped-in from host cache</td></tr>
<tr><td>Swap in from host cache</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of memory swapped-in from host cache</td></tr>
<tr><td>Swap in from host cache</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of memory swapped-in from host cache</td></tr>
<tr><td>Swap in rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate at which memory is swapped from disk into active memory during the interval</td></tr>
<tr><td>Swap in rate from host cache</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate at which memory is being swapped from host cache into active memory</td></tr>
<tr><td>Swap out</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of memory swapped-out to disk</td></tr>
<tr><td>Swap out</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory swapped-out to disk</td></tr>
<tr><td>Swap out</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of memory swapped-out to disk</td></tr>
<tr><td>Swap out</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of memory swapped-out to disk</td></tr>
<tr><td>Swap out rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate at which memory is being swapped from active memory to disk during the current interval</td></tr>
<tr><td>Swap out rate to host cache</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate at which memory is being swapped from active memory to host cache</td></tr>
<tr><td>Swap out to host cache</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of memory swapped-out to host cache</td></tr>
<tr><td>Swap out to host cache</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory swapped-out to host cache</td></tr>
<tr><td>Swap out to host cache</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of memory swapped-out to host cache</td></tr>
<tr><td>Swap out to host cache</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of memory swapped-out to host cache</td></tr>
<tr><td>Swap target</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Target size for the virtual machine swap file</td></tr>
<tr><td>Swap target</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Target size for the virtual machine swap file</td></tr>
<tr><td>Swap target</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Target size for the virtual machine swap file</td></tr>
<tr><td>Swap target</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Target size for the virtual machine swap file</td></tr>
<tr><td>Swap unreserved</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of memory that is unreserved by swap</td></tr>
<tr><td>Swap unreserved</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory that is unreserved by swap</td></tr>
<tr><td>Swap unreserved</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of memory that is unreserved by swap</td></tr>
<tr><td>Swap unreserved</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of memory that is unreserved by swap</td></tr>
<tr><td>Swap used</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of memory that is used by swap</td></tr>
<tr><td>Swap used</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory that is used by swap</td></tr>
<tr><td>Swap used</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of memory that is used by swap</td></tr>
<tr><td>Swap used</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of memory that is used by swap</td></tr>
<tr><td>Swapped</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel</td></tr>
<tr><td>Swapped</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel</td></tr>
<tr><td>Swapped</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel</td></tr>
<tr><td>Swapped</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel</td></tr>
<tr><td>Total</td>
<td>MB</td>
<td>absolute</td>
<td>average</td>
<td>Total amount of machine memory of all hosts in the cluster that is available for virtual machine memory (physical memory for use by the Guest OS) and virtual machine overhead memory</td></tr>
<tr><td>Total capacity</td>
<td>MB</td>
<td>absolute</td>
<td>average</td>
<td>Total amount of memory reservation used by and available for powered-on virtual machines and vSphere services on the host</td></tr>
<tr><td>Unreserved</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of memory that is unreserved</td></tr>
<tr><td>Unreserved</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of memory that is unreserved</td></tr>
<tr><td>Unreserved</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of memory that is unreserved</td></tr>
<tr><td>Unreserved</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of memory that is unreserved</td></tr>
<tr><td>Usage</td>
<td>Percent</td>
<td>absolute</td>
<td>none</td>
<td>Memory usage as percentage of total configured or available memory</td></tr>
<tr><td>Usage</td>
<td>Percent</td>
<td>absolute</td>
<td>average</td>
<td>Memory usage as percentage of total configured or available memory</td></tr>
<tr><td>Usage</td>
<td>Percent</td>
<td>absolute</td>
<td>minimum</td>
<td>Memory usage as percentage of total configured or available memory</td></tr>
<tr><td>Usage</td>
<td>Percent</td>
<td>absolute</td>
<td>maximum</td>
<td>Memory usage as percentage of total configured or available memory</td></tr>
<tr><td>Used by VMkernel</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses</td></tr>
<tr><td>Used by VMkernel</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses</td></tr>
<tr><td>Used by VMkernel</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses</td></tr>
<tr><td>Used by VMkernel</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses</td></tr>
<tr><td>Worst case allocation</td>
<td>MB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory allocation as calculated by the VMkernel scheduler based on current estimated demand and reservation, limit, and shares policies set for all virtual machines and resource pools in the host or cluster</td></tr>
<tr><td>Zero</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>Memory that contains 0s only</td></tr>
<tr><td>Zero</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Memory that contains 0s only</td></tr>
<tr><td>Zero</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>Memory that contains 0s only</td></tr>
<tr><td>Zero</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>Memory that contains 0s only</td></tr>
<tr><td>Zipped memory</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory (KB) zipped</td></tr>
<tr><td>swapIn</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>swapIn</td></tr>
<tr><td>swapIn</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>swapIn</td></tr>
<tr><td>swapIn</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>swapIn</td></tr>
<tr><td>swapIn</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>swapIn</td></tr>
<tr><td>swapOut</td>
<td>KB</td>
<td>absolute</td>
<td>none</td>
<td>swapOut</td></tr>
<tr><td>swapOut</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>swapOut</td></tr>
<tr><td>swapOut</td>
<td>KB</td>
<td>absolute</td>
<td>minimum</td>
<td>swapOut</td></tr>
<tr><td>swapOut</td>
<td>KB</td>
<td>absolute</td>
<td>maximum</td>
<td>swapOut</td></tr>
<tr><td>userworld</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>userworld</td></tr>
<tr><td>userworld</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>userworld</td></tr>
<tr><td>vm</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>vm</td></tr>
<tr><td>vm</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>vm</td></tr>
<tr><td>vmOvhd</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>vmOvhd</td></tr>
<tr><td>vmOvrhd</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>vmOvrhd</td></tr>
<tr><td>vmkOvrhd</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>vmkOvrhd</td></tr>
<tr><td>vmkOvrhd</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>vmkOvrhd</td></tr>
</table>
<hr/>
<h3>Network</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Broadcast receives</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of broadcast packets received during the sampling interval</td></tr>
<tr><td>Broadcast transmits</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of broadcast packets transmitted during the sampling interval</td></tr>
<tr><td>Data receive rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average rate at which data was received during the interval</td></tr>
<tr><td>Data receive rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average amount of data received per second</td></tr>
<tr><td>Data transmit rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average rate at which data was transmitted during the interval</td></tr>
<tr><td>Data transmit rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average amount of data transmitted per second</td></tr>
<tr><td>Multicast receives</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of multicast packets received during the sampling interval</td></tr>
<tr><td>Multicast transmits</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of multicast packets transmitted during the sampling interval</td></tr>
<tr><td>Packet receive errors</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of packets with errors received during the sampling interval</td></tr>
<tr><td>Packet transmit errors</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of packets with errors transmitted during the sampling interval</td></tr>
<tr><td>Packets received</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of packets received during the interval</td></tr>
<tr><td>Packets transmitted</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of packets transmitted during the interval</td></tr>
<tr><td>Receive packets dropped</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of receives dropped</td></tr>
<tr><td>Transmit packets dropped</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of transmits dropped</td></tr>
<tr><td>Unknown protocol frames</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of frames with unknown protocol received during the sampling interval</td></tr>
<tr><td>Usage</td>
<td>KBps</td>
<td>rate</td>
<td>none</td>
<td>Network utilization (combined transmit-rates and receive-rates) during the interval</td></tr>
<tr><td>Usage</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Network utilization (combined transmit-rates and receive-rates) during the interval</td></tr>
<tr><td>Usage</td>
<td>KBps</td>
<td>rate</td>
<td>minimum</td>
<td>Network utilization (combined transmit-rates and receive-rates) during the interval</td></tr>
<tr><td>Usage</td>
<td>KBps</td>
<td>rate</td>
<td>maximum</td>
<td>Network utilization (combined transmit-rates and receive-rates) during the interval</td></tr>
<tr><td>pNic Packets Received and Transmitted per Second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average rate of packets received and transmitted per second</td></tr>
<tr><td>pNic Throughput Provisioned</td>
<td>KBps</td>
<td>absolute</td>
<td>average</td>
<td>Provisioned pNic I/O Throughput</td></tr>
<tr><td>pNic Throughput Usable</td>
<td>KBps</td>
<td>absolute</td>
<td>average</td>
<td>Usable pNic I/O Throughput</td></tr>
<tr><td>pNic Throughput Usage for FT</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average pNic I/O rate for FT</td></tr>
<tr><td>pNic Throughput Usage for NFS</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average pNic I/O rate for NFS</td></tr>
<tr><td>pNic Throughput Usage for VMs</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average pNic I/O rate for VMs</td></tr>
<tr><td>pNic Throughput Usage for VR</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average pNic I/O rate for VR</td></tr>
<tr><td>pNic Throughput Usage for iSCSI</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average pNic I/O rate for iSCSI</td></tr>
<tr><td>pNic Throughput Usage for vMotion</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average pNic I/O rate for vMotion</td></tr>
<tr><td>vNic Throughput Contention</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Count of vNic packet drops</td></tr>
<tr><td>vNic Throughput Usage</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average vNic I/O rate</td></tr>
</table>
<hr/>
<h3>Power</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Cap</td>
<td>Watt</td>
<td>absolute</td>
<td>average</td>
<td>Maximum allowed power usage</td></tr>
<tr><td>Energy usage</td>
<td>Joule</td>
<td>delta</td>
<td>summation</td>
<td>Total energy used since last stats reset</td></tr>
<tr><td>Host Power Capacity Provisioned</td>
<td>Percent</td>
<td>absolute</td>
<td>average</td>
<td>Current power usage as a percentage of maximum allowed power</td></tr>
<tr><td>Host Power Capacity Usable</td>
<td>Watt</td>
<td>absolute</td>
<td>average</td>
<td>Current maximum allowed power usage</td></tr>
<tr><td>Power Capacity Usage</td>
<td>Watt</td>
<td>absolute</td>
<td>average</td>
<td>Current power usage</td></tr>
<tr><td>Usage</td>
<td>Watt</td>
<td>rate</td>
<td>average</td>
<td>Current power usage</td></tr>
</table>
<hr/>
<h3>Resource group CPU</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Active (1 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU active average over 1 minute</td></tr>
<tr><td>Active (1 min. peak)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU active peak over 1 minute</td></tr>
<tr><td>Active (15 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU active average over 15 minutes</td></tr>
<tr><td>Active (15 min. peak)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU active peak over 15 minutes</td></tr>
<tr><td>Active (5 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU active average over 5 minutes</td></tr>
<tr><td>Active (5 min. peak)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU active peak over 5 minutes</td></tr>
<tr><td>Group CPU sample count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Group CPU sample count</td></tr>
<tr><td>Group CPU sample period</td>
<td>Millisecond</td>
<td>absolute</td>
<td>latest</td>
<td>Group CPU sample period</td></tr>
<tr><td>Running (1 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU running average over 1 minute</td></tr>
<tr><td>Running (1 min. peak)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU running peak over 1 minute</td></tr>
<tr><td>Running (15 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU running average over 15 minutes</td></tr>
<tr><td>Running (15 min. peak)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU running peak over 15 minutes</td></tr>
<tr><td>Running (5 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU running average over 5 minutes</td></tr>
<tr><td>Running (5 min. peak)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU running peak over 5 minutes</td></tr>
<tr><td>Throttled (1 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>Amount of CPU resources over the limit that were refused, average over 1 minute</td></tr>
<tr><td>Throttled (15 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>Amount of CPU resources over the limit that were refused, average over 15 minutes</td></tr>
<tr><td>Throttled (5 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>Amount of CPU resources over the limit that were refused, average over 5 minutes</td></tr>
</table>
<hr/>
<h3>Storage adapter</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Average commands issued per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of commands issued per second by the storage adapter during the collection interval</td></tr>
<tr><td>Average read requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of read commands issued per second by the storage adapter during the collection interval</td></tr>
<tr><td>Average write requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of write commands issued per second by the storage adapter during the collection interval</td></tr>
<tr><td>Highest latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>latest</td>
<td>Highest latency value across all storage adapters used by the host</td></tr>
<tr><td>Read latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>The average time a read by the storage adapter takes</td></tr>
<tr><td>Read rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of reading data by the storage adapter</td></tr>
<tr><td>Storage Adapter Number Queued</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>The current number of I/Os that are waiting to be issued</td></tr>
<tr><td>Storage Adapter Outstanding I/Os</td>
<td>Percent</td>
<td>absolute</td>
<td>average</td>
<td>The percent of I/Os that have been issued but have not yet completed</td></tr>
<tr><td>Storage Adapter Outstanding I/Os</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>The number of I/Os that have been issued but have not yet completed</td></tr>
<tr><td>Storage Adapter Queue Command Latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval</td></tr>
<tr><td>Storage Adapter Queue Depth</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>The maximum number of I/Os that can be outstanding at a given time</td></tr>
<tr><td>Storage Adapter Throughput Contention</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time for an I/O operation to complete</td></tr>
<tr><td>Storage Adapter Throughput Usage</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>The storage adapter's I/O rate</td></tr>
<tr><td>Write latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>The average time a write by the storage adapter takes</td></tr>
<tr><td>Write rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of writing data by the storage adapter</td></tr>
</table>
<hr/>
<h3>Storage path</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Average commands issued per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of commands issued per second on the storage path during the collection interval</td></tr>
<tr><td>Average read requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of read commands issued per second on the storage path during the collection interval</td></tr>
<tr><td>Average write requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of write commands issued per second on the storage path during the collection interval</td></tr>
<tr><td>Highest latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>latest</td>
<td>Highest latency value across all storage paths used by the host</td></tr>
<tr><td>Read latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>The average time a read issued on the storage path takes</td></tr>
<tr><td>Read rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of reading data on the storage path</td></tr>
<tr><td>Storage Path Bus Resets</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of SCSI-bus reset commands issued during the collection interval</td></tr>
<tr><td>Storage Path Command Aborts</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of SCSI commands aborted during the collection interval</td></tr>
<tr><td>Storage Path Throughput Contention</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time for an I/O operation to complete</td></tr>
<tr><td>Storage Path Throughput Usage</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Storage path I/O rate</td></tr>
<tr><td>Write latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>The average time a write issued on the storage path takes</td></tr>
<tr><td>Write rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of writing data on the storage path</td></tr>
</table>
<hr/>
<h3>System</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Disk usage</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>Amount of disk space usage for each mount point</td></tr>
<tr><td>Heartbeat</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>Number of heartbeats issued per virtual machine during the interval</td></tr>
<tr><td>Heartbeat</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of heartbeats issued per virtual machine during the interval</td></tr>
<tr><td>OS Uptime</td>
<td>Second</td>
<td>absolute</td>
<td>latest</td>
<td>Total time elapsed, in seconds, since last operating system boot-up</td></tr>
<tr><td>Resource CPU active (1 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU active average over 1 minute of the system resource group</td></tr>
<tr><td>Resource CPU active (5 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU active average over 5 minutes of the system resource group</td></tr>
<tr><td>Resource CPU allocation maximum, in MHz</td>
<td>MHz</td>
<td>absolute</td>
<td>latest</td>
<td>CPU allocation limit, in MHz, of the system resource group</td></tr>
<tr><td>Resource CPU allocation minimum, in MHz</td>
<td>MHz</td>
<td>absolute</td>
<td>latest</td>
<td>CPU allocation reservation, in MHz, of the system resource group</td></tr>
<tr><td>Resource CPU allocation shares</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>CPU allocation shares of the system resource group</td></tr>
<tr><td>Resource CPU maximum limited (1 min.)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU maximum limited over 1 minute of the system resource group</td></tr>
<tr><td>Resource CPU maximum limited (5 min.)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU maximum limited over 5 minutes of the system resource group</td></tr>
<tr><td>Resource CPU running (1 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU running average over 1 minute of the system resource group</td></tr>
<tr><td>Resource CPU running (5 min. average)</td>
<td>Percent</td>
<td>absolute</td>
<td>latest</td>
<td>CPU running average over 5 minutes of the system resource group</td></tr>
<tr><td>Resource CPU usage (Average)</td>
<td>MHz</td>
<td>rate</td>
<td>average</td>
<td>Amount of CPU used by the Service Console and other applications during the interval</td></tr>
<tr><td>Resource CPU usage (Maximum)</td>
<td>MHz</td>
<td>rate</td>
<td>maximum</td>
<td>Amount of CPU used by the Service Console and other applications during the interval</td></tr>
<tr><td>Resource CPU usage (Minimum)</td>
<td>MHz</td>
<td>rate</td>
<td>minimum</td>
<td>Amount of CPU used by the Service Console and other applications during the interval</td></tr>
<tr><td>Resource CPU usage (None)</td>
<td>MHz</td>
<td>rate</td>
<td>none</td>
<td>Amount of CPU used by the Service Console and other applications during the interval</td></tr>
<tr><td>Resource memory allocation maximum, in KB</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory allocation limit, in KB, of the system resource group</td></tr>
<tr><td>Resource memory allocation minimum, in KB</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory allocation reservation, in KB, of the system resource group</td></tr>
<tr><td>Resource memory allocation shares</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Memory allocation shares of the system resource group</td></tr>
<tr><td>Resource memory mapped</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory mapped by the system resource group</td></tr>
<tr><td>Resource memory overhead</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Overhead memory consumed by the system resource group</td></tr>
<tr><td>Resource memory share saved</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory saved due to sharing by the system resource group</td></tr>
<tr><td>Resource memory shared</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory shared by the system resource group</td></tr>
<tr><td>Resource memory swapped</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory swapped out by the system resource group</td></tr>
<tr><td>Resource memory touched</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Memory touched by the system resource group</td></tr>
<tr><td>Resource memory zero</td>
<td>KB</td>
<td>absolute</td>
<td>latest</td>
<td>Zero filled memory used by the system resource group</td></tr>
<tr><td>Uptime</td>
<td>Second</td>
<td>absolute</td>
<td>latest</td>
<td>Total time elapsed, in seconds, since last system startup</td></tr>
</table>
<hr/>
<h3>Virtual disk</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Average number of outstanding read requests</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Average number of outstanding read requests to the virtual disk during the collection interval</td></tr>
<tr><td>Average number of outstanding write requests</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Average number of outstanding write requests to the virtual disk during the collection interval</td></tr>
<tr><td>Average read requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of read commands issued per second to the virtual disk during the collection interval</td></tr>
<tr><td>Average write requests per second</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Average number of write commands issued per second to the virtual disk during the collection interval</td></tr>
<tr><td>Read latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>The average time a read from the virtual disk takes</td></tr>
<tr><td>Read rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of reading data from the virtual disk</td></tr>
<tr><td>Read workload metric</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS virtual disk metric for the read workload model</td></tr>
<tr><td>Virtual Disk Throughput Contention</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>Average amount of time for an I/O operation to complete</td></tr>
<tr><td>Virtual Disk Throughput Usage</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Virtual disk I/O rate</td></tr>
<tr><td>Write latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>average</td>
<td>The average time a write to the virtual disk takes</td></tr>
<tr><td>Write rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Rate of writing data to the virtual disk</td></tr>
<tr><td>Write workload metric</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Storage DRS virtual disk metric for the write workload model</td></tr>
<tr><td>busResets</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>busResets</td></tr>
<tr><td>commandsAborted</td>
<td>Number</td>
<td>delta</td>
<td>summation</td>
<td>commandsAborted</td></tr>
</table>
<hr/>
<h3>Virtual machine operations</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Storage vMotion count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of migrations with Storage vMotion (datastore change operations for powered-on VMs)</td></tr>
<tr><td>VM clone count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine clone operations</td></tr>
<tr><td>VM create count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine create operations</td></tr>
<tr><td>VM datastore change count (non-powered-on VMs)</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of datastore change operations for powered-off and suspended virtual machines</td></tr>
<tr><td>VM delete count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine delete operations</td></tr>
<tr><td>VM guest reboot count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine guest reboot operations</td></tr>
<tr><td>VM guest shutdown count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine guest shutdown operations</td></tr>
<tr><td>VM host and datastore change count (non-powered-on VMs)</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of host and datastore change operations for powered-off and suspended virtual machines</td></tr>
<tr><td>VM host change count (non-powered-on VMs)</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of host change operations for powered-off and suspended VMs</td></tr>
<tr><td>VM power off count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine power off operations</td></tr>
<tr><td>VM power on count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine power on operations</td></tr>
<tr><td>VM reconfigure count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine reconfigure operations</td></tr>
<tr><td>VM register count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine register operations</td></tr>
<tr><td>VM reset count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine reset operations</td></tr>
<tr><td>VM standby guest count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine standby guest operations</td></tr>
<tr><td>VM suspend count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine suspend operations</td></tr>
<tr><td>VM template deploy count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine template deploy operations</td></tr>
<tr><td>VM unregister count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of virtual machine unregister operations</td></tr>
<tr><td>vMotion count</td>
<td>Number</td>
<td>absolute</td>
<td>latest</td>
<td>Number of migrations with vMotion (host change operations for powered-on VMs)</td></tr>
</table>
<hr/>
<h3>vCenter debugging information</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Activation count</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>Activation operations in vCenter</td></tr>
<tr><td>Activation count</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>Activation operations in vCenter</td></tr>
<tr><td>Activation count</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>Activation operations in vCenter</td></tr>
<tr><td>Activation latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>maximum</td>
<td>The latency of an activation operation in vCenter</td></tr>
<tr><td>Activation latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>minimum</td>
<td>The latency of an activation operation in vCenter</td></tr>
<tr><td>Activation latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>summation</td>
<td>The latency of an activation operation in vCenter</td></tr>
<tr><td>Host sync count</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>The number of host sync operations in vCenter</td></tr>
<tr><td>Host sync count</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>The number of host sync operations in vCenter</td></tr>
<tr><td>Host sync count</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>The number of host sync operations in vCenter</td></tr>
<tr><td>Host sync latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>maximum</td>
<td>The latency of a host sync operation in vCenter</td></tr>
<tr><td>Host sync latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>minimum</td>
<td>The latency of a host sync operation in vCenter</td></tr>
<tr><td>Host sync latency</td>
<td>Millisecond</td>
<td>absolute</td>
<td>summation</td>
<td>The latency of a host sync operation in vCenter</td></tr>
<tr><td>Inventory statistics</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>vCenter inventory statistics</td></tr>
<tr><td>Inventory statistics</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>vCenter inventory statistics</td></tr>
<tr><td>Inventory statistics</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>vCenter inventory statistics</td></tr>
<tr><td>Locking statistics</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>vCenter locking statistics</td></tr>
<tr><td>Locking statistics</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>vCenter locking statistics</td></tr>
<tr><td>Locking statistics</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>vCenter locking statistics</td></tr>
<tr><td>Managed object reference statistics</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>Managed object reference counts in vCenter</td></tr>
<tr><td>Managed object reference statistics</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>Managed object reference counts in vCenter</td></tr>
<tr><td>Managed object reference statistics</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>Managed object reference counts in vCenter</td></tr>
<tr><td>Miscellaneous</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>Miscellaneous statistics</td></tr>
<tr><td>Miscellaneous</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>Miscellaneous statistics</td></tr>
<tr><td>Miscellaneous</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>Miscellaneous statistics</td></tr>
<tr><td>Scoreboard statistics</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>Object counts in vCenter</td></tr>
<tr><td>Scoreboard statistics</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>Object counts in vCenter</td></tr>
<tr><td>Scoreboard statistics</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>Object counts in vCenter</td></tr>
<tr><td>Session statistics</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>The statistics of client sessions connected to vCenter</td></tr>
<tr><td>Session statistics</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>The statistics of client sessions connected to vCenter</td></tr>
<tr><td>Session statistics</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>The statistics of client sessions connected to vCenter</td></tr>
<tr><td>System statistics</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>The statistics of vCenter as a running system such as thread statistics and heap statistics</td></tr>
<tr><td>System statistics</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>The statistics of vCenter as a running system such as thread statistics and heap statistics</td></tr>
<tr><td>System statistics</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>The statistics of vCenter as a running system such as thread statistics and heap statistics</td></tr>
<tr><td>vCenter LRO statistics</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>vCenter LRO statistics</td></tr>
<tr><td>vCenter LRO statistics</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>vCenter LRO statistics</td></tr>
<tr><td>vCenter LRO statistics</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>vCenter LRO statistics</td></tr>
<tr><td>vCenter service statistics</td>
<td>Number</td>
<td>absolute</td>
<td>maximum</td>
<td>vCenter service statistics such as events, alarms, and tasks</td></tr>
<tr><td>vCenter service statistics</td>
<td>Number</td>
<td>absolute</td>
<td>minimum</td>
<td>vCenter service statistics such as events, alarms, and tasks</td></tr>
<tr><td>vCenter service statistics</td>
<td>Number</td>
<td>absolute</td>
<td>summation</td>
<td>vCenter service statistics such as events, alarms, and tasks</td></tr>
</table>
<hr/>
<h3>vCenter resource usage information</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>CPU privileged</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>CPU used by vCenter in privileged mode</td></tr>
<tr><td>CPU process</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>Total CPU used by vCenter</td></tr>
<tr><td>CPU queue length</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>Processor queue length on the system where vCenter is running</td></tr>
<tr><td>CPU system</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>Total system CPU used on the system where vCenter in running</td></tr>
<tr><td>CPU user</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>CPU used by vCenter in user mode</td></tr>
<tr><td>Context switch rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Number of context switches per second on the system where vCenter is running</td></tr>
<tr><td>Disk bytes read rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Number of bytes read from the disk per second on the system where vCenter is running</td></tr>
<tr><td>Disk bytes written rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Number of bytes written to the disk per second on the system where vCenter is running</td></tr>
<tr><td>Disk queue length</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>Disk queue length on the system where vCenter is running</td></tr>
<tr><td>Disk read rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Number of disk reads per second on the system where vCenter is running</td></tr>
<tr><td>Disk write rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Number of disk writes per second on the system where vCenter is running</td></tr>
<tr><td>Network queue length</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>Network queue length on the system where vCenter is running</td></tr>
<tr><td>Network usage</td>
<td>Percent</td>
<td>rate</td>
<td>average</td>
<td>Total network bytes received and sent per second on the system where vCenter is running</td></tr>
<tr><td>Page fault rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Number of page faults per second on the system where vCenter is running</td></tr>
<tr><td>Physical memory</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Physical memory used by vCenter</td></tr>
<tr><td>Pool non-paged bytes</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Memory pooled for non-paged bytes on the system where vCenter is running</td></tr>
<tr><td>Pool paged bytes</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Memory pooled for paged bytes on the system where vCenter is running</td></tr>
<tr><td>Process handles</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>Handles used by vCenter</td></tr>
<tr><td>Process threads</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>Number of threads used by vCenter</td></tr>
<tr><td>Received packet rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Rate of the number of total packets received per second on the system where vCenter is running</td></tr>
<tr><td>Sent packet rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Number of total packets sent per second on the system where vCenter is running</td></tr>
<tr><td>System call rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Number of systems calls made per second on the system where vCenter is running</td></tr>
<tr><td>System threads</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>Number of threads on the system where vCenter is running</td></tr>
<tr><td>Total packet rate</td>
<td>Number</td>
<td>rate</td>
<td>average</td>
<td>Number of total packets sent and received per second on the system where vCenter is running</td></tr>
<tr><td>Virtual memory</td>
<td>KB</td>
<td>absolute</td>
<td>average</td>
<td>Virtual memory used by vCenter</td></tr>
</table>
<hr/>
<h3>vSphere Replication</h3>
<table><thead><th>Metric</th><th>Unit</th><th>Stats Type</th><th>Rollup</th><th>Description</th></thead>
<tr><td>Replication Data Receive Rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average amount of data received per second</td></tr>
<tr><td>Replication Data Transmit Rate</td>
<td>KBps</td>
<td>rate</td>
<td>average</td>
<td>Average amount of data transmitted per second</td></tr>
<tr><td>vSphere Replication VM Count</td>
<td>Number</td>
<td>absolute</td>
<td>average</td>
<td>Current Number of Replicated VMs</td></tr>
</table>
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com2tag:blogger.com,1999:blog-18281936.post-15444308795817842732013-02-19T06:00:00.000-08:002013-02-19T06:00:03.758-08:00Exploring VMware vSphere PowerCLI with F#<p>I was recently asked to vet some VMware cluster capacity numbers.
It seemed like a task that might be repeated in the future and I really hate to manually
transcribe the data and compare them. So that means I need to write a script to automate it.
Fortunately, VMware has <a href="http://www.vmware.com/support/developer/PowerCLI/index.html">vSphere PowerCLI</a>
for this job. Unfortunately, the documentation for it was rather sparse. I also looked
at the book <a href="http://www.amazon.com/exec/obidos/ASIN/0470890797/techie2biz-20">
VMware vSphere PowerCLI Reference: Automating vSphere Administration</a>,
but really did not want to script
in PowerShell simply because I'm not that familiar with PowerShell.
However, PowerCLI is accessible from .NET also, so I can write my script in F#.</p>
<p>
<p>The following lines of script simply drills down from <code>Datacenter</code>
object to the <code>VirtualMachine</code> object. Once you grab all the objects,
you can explore the properties of each object.</p>
<pre class="brush: fsharp">
#r @"C:\pkg\VMware\Infrastructure\vSphere PowerCLI\VMware.Vim.dll"
open System
open VMware.Vim
open System.Collections.Specialized
let serviceUrl = "https://myVSphereHost/sdk"
let userId = "someUserId"
let password="somePassword"
let client = new VimClient()
let service = client.Connect(serviceUrl)
// Must login to do anything - if you are getting null values, it means the session automatically timed out
client.Login(userId,password)
// Let us get all the datacenters
let dataCenters = client.FindEntityViews(typeof<Datacenter>,null,null,null)
// Drill down into the first datacenter
let dc = dataCenters |> Seq.cast<Datacenter> |> Seq.head
// Get a cluster
let cluster = client.GetView(dc.Parent,null) :?> ClusterComputeResource
// Get the first host in the cluster
let host = client.GetView(cluster.Host |> Seq.head,null) :?> HostSystem
// Get the first VM on the physical host
let vm = client.GetView(host.Vm |> Seq.head,null) :?> VirtualMachine
</pre>
<p>Let's do something interesting with PowerCLI. In the following scripts,
I wanted to grab the capacity information at the VMware cluster level
and the combine allocation/utilization info of all the virtual machines hosted on the
VMware cluster.
<pre class="brush: fsharp">
// Utility function to help use get vSphere Entities
let getEntityViews viewType searchParams =
match searchParams with
| Some(searchParams0) ->
let filters = new NameValueCollection()
searchParams0 |> Seq.iter (fun (k,v) -> filters.Add(k,v))
client.FindEntityViews(viewType,null,filters,null)
| None ->
client.FindEntityViews(viewType,null,null,null)
// Get vSphere Entities with specific properties (reduces returned data)
// Don't know how to define a function with multiple arities in F# - clumsy workaround
let getEntityViews2 viewType searchParams props =
match searchParams with
| Some(searchParams0) ->
let filters = new NameValueCollection()
searchParams0 |> Seq.iter (fun (k,v) -> filters.Add(k,v))
client.FindEntityViews(viewType,null,filters,props)
| None ->
client.FindEntityViews(viewType,null,null,props)
// Get Cluster usage summary!
let getUsageSummary clusterName =
let clusterProps = [|"Summary"; "Host"|]
let hostProps = [|"Vm";"Name";"Hardware";"Runtime"|]
let vmProps = [|"Name";"Config";"Runtime";"Summary";"ResourceConfig"|]
let toMB memoryInBytes = memoryInBytes / (1024L*1024L)
// Get cluster - Expect only one result
printfn "Getting cluster data..."
let cluster =
let filters = Some([("name",clusterName)])
getEntityViews2 typeof<ClusterComputeResource> filters clusterProps
|> Seq.cast<ClusterComputeResource>
|> Seq.head
// Get all Hosts for this cluster
printfn "Getting host list..."
let hostList =
cluster.Host
|> Seq.map(fun moRef -> client.GetView(moRef,hostProps))
|> Seq.cast<HostSystem>
|> Seq.cache
printfn "Getting VM list..."
let vmList =
hostList
|> Seq.map (fun host -> host.Vm)
|> Seq.concat
|> Seq.map (fun moRef -> client.GetView(moRef,vmProps))
|> Seq.cast<VirtualMachine>
|> Seq.cache
let clusterCores = cluster.Summary.NumCpuCores
let clusterCPU = cluster.Summary.TotalCpu
let clusterMemory = cluster.Summary.TotalMemory
// Utility function to get summation results from selected fields
let inline total (extractor:VirtualMachine -> Nullable<'b>) =
vmList
|> Seq.map extractor
|> Seq.map (fun x -> x.GetValueOrDefault())
|> Seq.sum
printfn "Getting Cluster Summary Info..."
let cpuUsed = total (fun vm -> vm.Summary.Config.NumCpu)
let memoryUsed = total (fun vm -> vm.Summary.Config.MemorySizeMB)
let cpuReserved = total (fun vm -> vm.Summary.Config.CpuReservation)
let memoryReserved = total (fun vm -> vm.Summary.Config.MemoryReservation)
let maxMemoryUsage = total (fun vm -> vm.Runtime.MaxMemoryUsage)
let maxCpuUsage = total (fun vm -> vm.Runtime.MaxCpuUsage)
let cpuReservation = total (fun vm -> vm.ResourceConfig.CpuAllocation.Reservation)
let cpuLimit = total (fun vm -> vm.ResourceConfig.CpuAllocation.Limit)
let memoryReservation = total (fun vm -> vm.ResourceConfig.MemoryAllocation.Reservation)
let memoryLimit = total (fun vm -> vm.ResourceConfig.MemoryAllocation.Limit)
printfn "Cluster Name : %s" clusterName
printfn "Number of Hosts in Cluster : %i" (Seq.length hostList)
printfn "Number of VMs in Cluster : %i" (Seq.length vmList)
printfn "Cluster Total Cores : %i" clusterCores
printfn "Cluster Total CPU (MHz) : %i" clusterCPU
printfn "Cluster Total Memory : %i" clusterMemory
printfn "Cluster Total Memory (MB) : %i" (toMB clusterMemory)
printfn "CPU Used by VMs : %i" cpuUsed
printfn "Memory Used by VMs : %i" memoryUsed
printfn "CPU Reserved by VMs : %i" cpuReserved
printfn "Memory Reserved by VMs : %i" memoryReserved
printfn "Max Memory Usage by VMs : %i" maxMemoryUsage
printfn "Max CPU Usage by VMs (MHz) : %i" maxCpuUsage
printfn "Total Allocated CPU Reservations : %i" cpuReservation
printfn "Total Allocated CPU Limits : %i" cpuLimit
printfn "Total Allocated Memory Reservations : %i" memoryReservation
printfn "Total Allocated Memory Limits : %i" memoryLimit
printfn "Memory Used / Cluster Total Memory : %f" (double(memoryUsed) / double(toMB clusterMemory))
printfn "vCPU Allocated / Cluster Total Cores : %f" (float(cpuUsed) / float(clusterCores))
printfn "Done!"
// Invoke getUsageSummary to get the summary info on my cluster
getUsageSummary "MyTestCluster"
</pre>
<p>Sample results:</p>
<pre class="brush: plain">
Number of Hosts in Cluster : 5
Number of VMs in Cluster : 10
Cluster Total Cores : 120
Cluster Total CPU (MHz) : 276000
Cluster Total Memory : 1374369136640
Cluster Total Memory (MB) : 1310700
CPU Used by VMs : 38
Memory Used by VMs : 253952
CPU Reserved by VMs : 0
Memory Reserved by VMs : 0
Max Memory Usage by VMs : 253952
Max CPU Usage by VMs (MHz) : 87400
Total Allocated CPU Reservations : 0
Total Allocated CPU Limits : -10
Total Allocated Memory Reservations : 0
Total Allocated Memory Limits : -10
Memory Used / Cluster Total Memory : 0.193753
vCPU Allocated / Cluster Total Cores : 0.316667
</pre>
<p>I wish VMware had PowerCLI class documentation similar to those for the .NET library hosted
on MSDN. If VMware does have those documentation, I can't seem to find them. Lack of documentation
has forced me to interactively explore the PowerCLI with F#.
Thankfully, I can do this in F# and
shudder at the thought of exploring PowerCLI in C#.</p>
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-84552778257978249272013-02-08T06:00:00.000-08:002013-02-08T06:00:03.328-08:00Clojure Script to Convert Java GC log to CSV Format<p>My <a href="http://jyliao.blogspot.com/2013/02/java-garbage-collection-log-analysis.html">
previous blog post</a> showed how to use R to perform Java GC log analysis. Ajit Joglekar asked that I post
the Clojure script. Here's the Clojure script that I used to convert GC log to CSV format.
I have not tested this with a wide variety of GC logs but I do know that this script will not work for CMS or G1GC gc logs.
Modify <code>minor-gc-pattern</code> and <code>full-gc-pattern</code> to match the gc log output as needed.
</p>
<pre class="brush: clojure">
(use '[clojure.string :only (join)])
; Match for pause time "0.1566980 secs]"
(def ^:constant pause-time "([\\d\\.]+) secs\\]")
; Match for Java heap space stat "524288K->32124K(2009792K)"
(def ^:constant space "(\\d+)K->(\\d+)K\\((\\d+)K\\)")
; Match for Execution stat "[Times: user=0.24 sys=0.06, real=0.16 secs]"
(def ^:constant exec-stat " \\[Times: user=([\\d\\.]+) sys=([\\d\\.]+), real=([\\d\\.]+) secs\\]")
; Example Minor GC entry
; 212.785: [GC [PSYoungGen: 524288K->32124K(611648K)] 524288K->32124K(2009792K), 0.1566980 secs] [Times: user=0.24 sys=0.06, real=0.16 secs]
; Define regex pattern to parse young gen GC event
(defn minor-gc-pattern []
(let [timestamp "([\\d\\.]+): \\[GC .*\\[PS.*: "
young-gen (str space "] ")
heap (str space ", ")]
(re-pattern (str timestamp young-gen heap pause-time exec-stat))))
; Example Full GC entry
;"43587.513: [Full GC (System) [PSYoungGen: 964K->0K(598912K)] [PSOldGen: 142673K->120674K(1398144K)] 143637K->120674K(1997056K) [PSPermGen: 82179K->82179K(147520K)], 0.7556570 secs] [Times: user=0.76 sys=0.00, real=0.77 secs]"
; Define the regex pattern to parse each line in gc log
(defn full-gc-pattern []
(let [timestamp "([\\d\\.]+): \\[Full.*"
young-gen (str ": " space "]")
old-gen (str " \\[\\w+: " space "\\] ")
perm-gen (str " \\[\\w+: "space "\\], ")
heap space]
(re-pattern (str timestamp
young-gen
old-gen
heap
perm-gen
pause-time
exec-stat))))
; Variable definitions (for both process-full-gc & process-minor-gc
; ts - timestamp (in seconds)
; ys - YoungGen space starting heap size (in KB)
; ye - YoungGen space ending heap size (in KB)
; ym - YoungGen space max heap size (in KB)
; os - OldGen space starting heap size (in KB)
; oe - OldGen space ending heap size (in KB)
; om - OldGen space max heap size (in KB)
; hs - Total heap space starting heap size (in KB)
; he - Total heap space ending heap size (in KB)
; hm - Total heap space max heap size (in KB)
; pt - GC Pause Time (in seconds)
; ps - PermGen space starting heap size (in KB)
; pe - PermGen space ending heap size (in KB)
; pm - PermGen space max heap size (in KB)
; ut - User Time (in seconds)
; kt - Kernel Time (in seconds)
; rt - Real Time (in seconds)
(defn process-full-gc [entry]
(let [[a ts ys ye ym os oe om hs he hm ps pe pm pt ut kt rt & e] entry]
(join \, [ts "full" pt
ys ye ym
hs he hm
ut kt rt
os oe om
ps pe pm])))
(defn process-minor-gc [entry]
(let [[a ts ys ye ym hs he hm pt ut kt rt & e] entry]
(join \, [ts "minor" pt
ys ye ym
hs he hm
ut kt rt])))
(def headers (join \, ["timestamp" "gc.type" "pause.time"
"young.start" "young.end" "young.max"
"heap.start" "heap.end" "heap.max"
"time.user" "time.sys" "time.real"
"old.start" "old.end" "old.max"
"perm.start" "perm.end" "perm.max"]))
(defn process-gc-file [infile outfile]
(let [gcdata (line-seq (clojure.java.io/reader (clojure.java.io/file infile)))]
(with-open [w (clojure.java.io/writer outfile)]
(let [writeln (fn [x] (.write w (str x "\n")))]
(writeln headers)
(doseq [line gcdata]
(let [minor-gc (re-seq (minor-gc-pattern) line)
full-gc (re-seq (full-gc-pattern) line)]
(when-not (nil? full-gc)
(writeln (process-full-gc (first full-gc))))
(when-not (nil? minor-gc)
(writeln (process-minor-gc (first minor-gc))))))))))
;-----------------------------------------------------------------------
; Convert Java GC log csv format
;-----------------------------------------------------------------------
(process-gc-file "gc.log" "data.csv")
</pre>
<p>You can download the script directly here : <a href="https://sites.google.com/site/programminglibrary/scripts/gcanalysis.clj">gcanalysis.clj</a>.
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com2tag:blogger.com,1999:blog-18281936.post-75357033350065828392013-02-04T05:00:00.000-08:002013-02-04T05:00:07.103-08:00Java Garbage Collection Log Analysis with R<p>In the past, I generally perform Java GC log analysis using a combination of scripts such as the one I have written in my previous <a href="http://jyliao.blogspot.com/2012/06/quick-and-easy-way-to-monitor-for.html">blog post</a> for parsing the gc log and transforming it into csv format, <a href="http://java.net/projects/gchisto">GCHisto</a> tool as mentioned in the book Java Performance, and Microsoft Excel for mostly plotting purposes. However, as I find myself doing more and more GC analysis for troubleshooting performance issues, I'm finding that these tools are not flexible enough for my purposes. </p>
<p>I heard about <a href="http://www.r-project.org/">The R Project</a> from one my coworker. In addition, I see R <a href="http://www.revolutionanalytics.com/what-is-open-source-r/companies-using-r.php">gaining traction</a> in the industry; I decided to try using R for GC analysis. This blog post is to document some of my experiences with using R for GC analysis.</p>
<p>There are some posts in the blogosphere that seem to suggest that it is difficult to learn R. I did not find that to be the case. Whether that's due to knowing functional programming or past experience with Matlab, I don't know. I did purchase a bunch of books on R to learn it including
<a href="http://www.amazon.com/exec/obidos/ASIN/144931208X/techie2biz-20">R in a Nutshell</a>,
<a href="http://www.amazon.com/exec/obidos/ASIN/1461413648/techie2biz-20">R by Example</a>,
<a href="http://www.amazon.com/exec/obidos/ASIN/1935182390/techie2biz-20">R in Action</a>,
<a href="http://www.amazon.com/exec/obidos/ASIN/0387790535/techie2biz-20">Introductory Statistics with R</a>.
I got the most mileage with the last book.</p>
<p>In working with R, I still would use a script to parse out raw GC log and converted into csv file. So far, most of my GC analysis are on non-CMS GC logs so this blog post is restricted to discussing analysis of mostly parallel GC logs. In order to be able to better read my R scripts, I have changed my script to output the csv file with the following headers:
<pre class="brush: plain">
timestamp,gc.type,young.start,young.end,young.max,heap.start,heap.end,heap.max,pause.time,old.start,old.end,old.max,perm.start,perm.end,perm.max,time.user,time.sys,time.real
</pre>
<p>I can load the csv file in R as follows:</p>
<pre class="brush: r">
data.file <- "gc.log.csv"
data.gc <- read.csv(data.file,sep=',')
</pre>
<p>I created the following utility functions because I find that I often want to work in MB or GB for heap space rather than in KB. Also, I want to work with timestamp in hours rather than seconds.</p>
<pre class="brush: r">
sec.to.hours <- function(x) x/(60.0*60.0)
kb.to.mb <- function(x) x/1024.0
kb.to.gb <- function(x) x/1024.0^2
</pre>
<h3>Printing GC Pause Time Statistics</h3>
<p>With this, I can dump the GC pause time statistical information as follows:</p>
<pre class="brush: r">
# Dump mean, standard deviation, GC counts for GC pause time
attach(data.gc)
by(pause.time,gc.type,mean)
by(pause.time,gc.type,sd)
by(pause.time,gc.type,length)
# GC Frequency statistics
summary(diff(timestamp))
# Time spent in GC
sum(pause.time)
# Percentage of time spent in GC
sum(pause.time)*100.0/max(timestamp)
detach(data.gc)
</pre>
<p>For the above code, a sample output would be:</p>
<pre class="brush: plain">
# by(pause.time,gc.type,mean) output
# GC Average Pause Time
gc.type: full
[1] 2.125116
--------------------------------------------------------------
gc.type: minor
[1] 0.06472383
# by(pause.time,gc.type,sd) output
# GC pause time distribution
gc.type: full
[1] 1.01972
--------------------------------------------------------------
gc.type: minor
[1] 0.02684703
# by(pause.time,gc.type,length) output
# GC count by type
gc.type: full
[1] 7
--------------------------------------------------------------
gc.type: minor
[1] 2172
# summary(diff(timestamp)) output
# GC Frequency statistics - (e.g. GC occurs on average every 84 seconds)
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.08 22.44 71.34 84.83 143.10 252.90
# sum(pause.time) output
# Total time spent in GC
[1] 155.456
# sum(pause.time)*100.0/max(timestamp) output
# Percentage of time spent in GC
[1] 0.08412511
</pre>
<h3>Plotting GC Timeline</h3>
<p>Here's how to plot GC Timeline:</p>
<pre class="brush: r">
with(data.gc, {
plot.title <-"GC Timeline"
gc.minor <- pause.time[gc.type == "minor"]
gc.full <- pause.time[gc.type == "full"]
# Plot bars for full GC pause time
plot(sec.to.hours(timestamp[gc.type == "full"]),
gc.full*1000.0,
xlab="Elapsed Time (hrs)",
ylab="GC Pause Time Time(ms)",type='h',
main=plot.title,col="red",ylim=c(0,1000),xlim=c(0,18))
# Plot bars for minor GC pause time
lines(sec.to.hours(timestamp[gc.type == "minor"]),
gc.minor*1000.0,col="dark green",type="h")
# Set tick marks for every hour
axis(1,seq(0,18,1))
grid()
})
</pre>
<p>A sample output of the above code would be:</p>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXQdLWql4zBykPqxFSh51oqty9oYu-sMkTrJPU-C6DN0McGKyld2JR3JtfivfW-ASBXU9_GWDTufkdAyKTU_XIYf23wtiPcaQRTGiXLjU0D0eyvSvn1oAw69iYpgsbLVpX9BkLNw/s1600/gctimeline.png" imageanchor="1" style=""><img border="0" height="178" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXQdLWql4zBykPqxFSh51oqty9oYu-sMkTrJPU-C6DN0McGKyld2JR3JtfivfW-ASBXU9_GWDTufkdAyKTU_XIYf23wtiPcaQRTGiXLjU0D0eyvSvn1oAw69iYpgsbLVpX9BkLNw/s400/gctimeline.png" /></a>
<h3>Plotting GC Pause Time Distribution</h3>
<p>To plot GC pause time distribution:</p>
<pre class="brush: r">
# Plot GC Pause Time Distribution
plot.title <- "GC Pause Time Distribution"
with(data.gc, {
hist(pause.time,breaks=100,
xlab="GC Pause Time(sec)",
xlim=c(0,1.0),
main=plot.title)
})
</pre>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibhvrL4FusvhUBKFgZVgogXIzQFeXj-FjsreVyg-_O-7zeDHPrNhz933M8icSY-Wt0PtyJTMWEmp3q2gNa8kID399452GcXbU64pDK01JyC_mVPUMrCunro-6U7Cdpgpqy6ok8XA/s1600/gchisto.png" imageanchor="1" style=""><img border="0" height="178" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibhvrL4FusvhUBKFgZVgogXIzQFeXj-FjsreVyg-_O-7zeDHPrNhz933M8icSY-Wt0PtyJTMWEmp3q2gNa8kID399452GcXbU64pDK01JyC_mVPUMrCunro-6U7Cdpgpqy6ok8XA/s400/gchisto.png" /></a>
<h3>Plotting GC Pause Time Distribution as a Boxplot</h3>
<p>Alternatively, you can view this as a boxplot:</p>
<pre class="brush: r">
plot.title <- "GC Pause Time Distribution as Boxplot"
with(data.gc, {
boxplot(pause.time ~ gc.type,main=plot.title)
})
</pre>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb5yRmX3i9onKW5izYd83LazF5cFbrnRpiV3NqrH9D0vjVtZcsgKsF_p-CuHW4ylPkvj8fJJf66cOe9XQQ6t2cXYeJVhn2LkhUPNiJt19vmiGLLivMLgx13rcXgwllYVxUhjb6QA/s1600/boxplot.png" imageanchor="1" style=""><img border="0" height="178" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb5yRmX3i9onKW5izYd83LazF5cFbrnRpiV3NqrH9D0vjVtZcsgKsF_p-CuHW4ylPkvj8fJJf66cOe9XQQ6t2cXYeJVhn2LkhUPNiJt19vmiGLLivMLgx13rcXgwllYVxUhjb6QA/s400/boxplot.png" /></a>
<h3>Plotting Promoted Heap Memory</h3>
<p>One particular case, I was analyzing a situation where
minor GC pause time was unusually large. My original hypothesis is that
a lot of memory was being promoted into old generation space. Therefore,
I wanted a plot of heap memory promoted into old gen space:</p>
<pre class="brush: r">
plot.title <- "Heap Promoted into Old Gen Space"
with(data.gc, {
old.end <- kb.to.mb(heap.end - young.end)
x <- sec.to.hours(timestamp)
# If there is a full GC, promoted heap size would go negative
y <- diff(old.end)
plot(x[-1],y,
xlab="Elapsed Time (hrs)",
ylab="Promoted Heap (MB)",type='h',
main=plot.title)
grid()
})
</pre>
<p>In this particular case, the promoted heap was not all that large as shown in the diagram below:</p>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYpCpriD2NhuqhS9_He34jDtyn-kI7RAUB-0Y85BnCgNAnAjoDZwwkLqa6J0Qz64X_BMt3TU8aL2gXLvbnQ0YxzwYWFK2nvQ0snrPqxPPlfc4D-EXBp0KiiHY2N6_qs9J9vWcR7Q/s1600/promotedheap.png" imageanchor="1" style=""><img border="0" height="178" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYpCpriD2NhuqhS9_He34jDtyn-kI7RAUB-0Y85BnCgNAnAjoDZwwkLqa6J0Qz64X_BMt3TU8aL2gXLvbnQ0YxzwYWFK2nvQ0snrPqxPPlfc4D-EXBp0KiiHY2N6_qs9J9vWcR7Q/s400/promotedheap.png" /></a>
<h3>Plotting Young Gen Survived Heap</h3>
<p>I then investigated survivor heap spaces. Here's the script that plots the survivor heap space along with GC pause time on the same chart:</p>
<pre class="brush: r">
# Plot survived Heap
xtitle <- 'Elapsed Time (hrs)'
ytitle <-'Survivor Heap (MB)'
plot.title <- "Survived Heap after Minor GC"
with(data.gc, {
x <- sec.to.hours(timestamp)
y <- kb.to.mb(young.end)
plot(x,y,
xlab=xtitle,ylab=ytitle,
main=plot.title,type='o',col='blue')
# Plot on secondary axis
par(new=TRUE)
plot(x,pause.time,
axes=FALSE,type='o',
col="green",pch=23,xlab="",ylab="")
axis(4)
mtext("GC Pause Time (sec)",4)
})
</pre>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj87VsugUQS4mJREtyYo9So__vVkuXvqyK2u-4qUrcle25898ySiepGse0gCJHDRnGfGvI1t5zoC30MxtL9Q1eogej24lhC9FFf_9-_rsheszcXZKZxoQLK3ClEaWuGPLGNlTeDyQ/s1600/survivorheap.png" imageanchor="1" style=""><img border="0" height="178" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj87VsugUQS4mJREtyYo9So__vVkuXvqyK2u-4qUrcle25898ySiepGse0gCJHDRnGfGvI1t5zoC30MxtL9Q1eogej24lhC9FFf_9-_rsheszcXZKZxoQLK3ClEaWuGPLGNlTeDyQ/s400/survivorheap.png" /></a>
<p>In addition, I can also check the correlation coefficient via :</p>
<pre class="brush: r">
> with(data.gc,cor(young.end[gc.type == 'minor'],pause.time[gc.type == 'minor']))
[1] 0.9659936
</pre>
<p>As we can see from the result, the correlation was pretty high that size of
the survivor heap and its effect on GC pause time.</p>
<h3>Checking for Memory Leaks</h3>
<p>Another plot that I often have to generate is used as a quick check for memory leaks.
Here's the R script that plots old gen heap space usage after every full GC:</p>
<pre class="brush: r">
# Check for memory leak
xtitle <- 'Elapsed Time (hrs)'
ytitle <-'Old Gen Heap (MB)'
with(data.gc, {
plot.title <- "Old Gen Space after Full GC"
x <- sec.to.hours(timestamp[gc.type=="full"])
y <- kb.to.mb(old.end[gc.type=="full"])
plot(x,y,
xlab=xtitle,ylab=ytitle,main=plot.title,type='o',col='blue')
# Add fitted line
abline(lm(y ~ x),col="red")
retval <<- lm(y ~ x)
})
retval
</pre>
<p>With the trend analysis line, we can potentially project when the system will run out of memory.
For the following plot, checking the linear model fit along with eyeballing the chart suggests that we don't really have a memory leak issue:</p>
<pre class="brush: plain">
Call:
lm(formula = y ~ x)
Coefficients:
(Intercept) x
1285.978 0.298
</pre>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1Ilulct52wcPktFIPS6JndGAWbY6gf-3Z9Hku8XHMgk0-amr_q8oujquQ63XMA2ZQ1SO9A4XLzFg5e2t9oSBaLLfPlJybv_9aMfvv36PXx_aa1IaH9bVRpOKmMS5mqCwF4mGBGg/s1600/memoryleak.png" imageanchor="1" style=""><img border="0" height="178" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1Ilulct52wcPktFIPS6JndGAWbY6gf-3Z9Hku8XHMgk0-amr_q8oujquQ63XMA2ZQ1SO9A4XLzFg5e2t9oSBaLLfPlJybv_9aMfvv36PXx_aa1IaH9bVRpOKmMS5mqCwF4mGBGg/s400/memoryleak.png" /></a>
<p>I am delighted by the power afforded by R to perform GC analysis. It has now become my preferred tool to perform GC analysis. In addition, because I learned how to use R, I'm beginning to use R in other capacity, including inventory , trending, and capacity planning analysis.
John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com6tag:blogger.com,1999:blog-18281936.post-91371066691774051242013-01-28T05:00:00.000-08:002013-01-28T05:00:09.862-08:00Checking Tibco EMS Queue Connected Users with Clojure<p>During our Tibco EMS infrastructure upgrade project, we had some rogue connections that just refused to disconnect
from the old infrastructure and switch to the new Tibco EMS infrastructure. In order to track down these connections
and evict those connections, I had to generate a report which would tell me which user, from which host is connected
to particular queues in Tibco EMS. Here's the script that I wrote in Clojure that generates that report.</p>
<pre class="brush: clojure;">
; Get destination users (and which host the user is connecting from) along with which queue/topic it is connected to.
(defn get-destination-users [server-url username password]
(with-open [admin (TibjmsAdmin. server-url username password)]
(let [connections (.getConnections admin)
consumers (.getConsumers admin)
dest-userids (->> consumers
(map (fn [c] {:userid (.getConnectionID c)
:dest (.getDestinationName c)})))
get-user (fn [dest-name id]
(let [conn (first (filter #(= id (.getID %)) connections))]
(if (nil? conn)
{:dest dest-name :user (str "Unknown user : " id) :host "unknown"}
{:dest dest-name :user (.getUserName conn) :host (.getHost conn)})))]
(map (fn [x] (get-user (:dest x) (:userid x))) dest-userids))))
; Dump results in CSV format
(def results (get-destination-users server-url username password))
(doseq [r (sort-by :user (set results))]
(println (str (:user r) "," (:host r) "," (:dest r))))
</pre>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-41366497136959022912013-01-14T04:00:00.000-08:002013-01-14T04:00:11.412-08:00Who's Connecting to My Servers?<p>I have been reading through the book
<a href="http://www.amazon.com/exec/obidos/ASIN/1449394701/techie2biz-20">Clojure Programming</a>.
In the course of reading this book, I've looked for all sorts of opportunities to try applying Clojure
at work. Most of the time, I've used Clojure to implement convenience scripts to
help with my day job. I would typically use Clojure whenever I have to work with Java based
platforms and F# for .NET based platforms. Occasionally, I would develop scripts that does not have
any dependency and I could choose any language to implement. What would typically happen
is that I would choose the programming language that I used last.
This strategy, unfortunately, would typically end up biasing me
toward one programming language and lately, it has been biasing
me toward Clojure.
After noticing this trend, I have decided to deliberately and consciously choose to implement
in the less frequently used language so I don't become completely rusty in the
other programming languages.
</p>
<p>Recently, I had to opportunity to write a small script. I was managing an infrastructure upgrade and needed to know the downstream impact. It was an infrastructure component that that a lot of persistent inbound connections, but unfortunately, the inbound connections were neither monitored nor documented. One way to check the connections is ask the the network engineers to setup monitoring
on the servers and collect the information on the incoming connections. Our network
engineers are generally pretty busy and we hate to add to their existing workloads. However,
we can effectively do the same thing by running <code>netstat -an</code> on each
of the target servers and taking that output dump and parse that for incoming connections.
We would do this over a period of time to try to capture most of the client connections.
</p>
<p>The following Clojure script loads all the <code>netstat</code> dump output files
and generate a list of all the hosts that are connected to the target servers:</p>
<pre class="brush: clojure;">
(import '(java.net InetAddress))
(use '[clojure.string :only (join)])
(use '[clojure.java.io :as io])
; Load all the data from all *.data files in c:\work\servers folder
(def data (->> "c:\\work\\servers"
(io/file)
(file-seq)
(map #(.getAbsolutePath %))
(filter #(re-matches #".*\.data$" %))
(map #(slurp %))
(join " ")))
; Find all ip addresses in the netstat dump
; Perform hostname lookup, discard duplicates, sort the hostnames
(def hosts (->> data
(re-seq #"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\.\d+")
(map #(second %))
(set)
(map #(.getCanonicalHostName (InetAddress/getByName %)))
(sort)
(join "\n")))
; Dump output to clients.out file
(spit "c:\\work\\servers\\results.out" hosts)
</pre>
<p>The above script runs with the assumption that all data fits into memory. However, if that becomes a
problem, it is fairly trivial to sequentially read and process <code>netstat</code> dump one file at a time
and combine the results to write to the output.</p>
<p>The F# version is similar to Clojure version. Grabbing the files from the folder is easier but the need to
explicitly handle exceptions adds back the additional lines of code to be about on par with code verbosity of the Clojure version.
</p>
<pre class="brush: fsharp;">
open System.IO
open System.Net
open System.Text.RegularExpressions
// Load all the data from all *.data files in c:\work\servers folder
let data = Directory.GetFiles(@"c:\work\servers","*.data")
|> Seq.map File.ReadAllText
|> String.concat " "
// Return hostname if it can be resolved
// otherwise return the ip address
let getHostEntry (ipaddress:string) =
try
Dns.GetHostEntry(ipaddress).HostName
with
| err -> ipaddress
// Find all ip addresses in the netstat dump
// Perform hostname lookup, discard duplicates, sort the hostnames
let hosts = Regex.Matches(data,@"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\.\d+")
|> Seq.cast<Match>
|> Seq.map (fun m -> m.Groups.[1].Value)
|> Set.ofSeq
|> Seq.map getHostEntry
|> Seq.sort
|> String.concat "\n"
File.WriteAllText(@"c:\work\servers\results.out",hosts)
</pre>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-87295390079914296782012-06-11T06:00:00.000-07:002012-06-11T06:00:05.908-07:00Quick and easy way to monitor for memory leaks in Java application servers<p>
I was part of a postmortem analysis team that performed a root cause analysis
on why a particular node in a HA paired system failed and needed to provide recommendations to
prevent future occurrences. Initial examination of the log showed that the node that failed had a permgen
out of memory problem. Management team also wanted to know if there were memory leaks
with the application.</p>
<p>
Unfortunately, the Java application server was not configured with GC logging nor configured to
perform a heap dump on out of memory error condition. Without the gc log data or the heap dump, it becomes harder to be able to explain why did the application server run out of permgen space at the time it failed. In the interest of collecting data and to prevent future node failures, we recommended that permgen space be increased and add the following startup parameters to the Java application server</p>
<pre>
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/myapp/heapdumps
-Xloggc:/myapp/logs/gc.log
</pre>
<p>One problem that arose during the writeup of the analysis was
how to answer the question whether the application
had any memory leak issue. The application team could not reproduce the problem in
non-production environment but wanted some assurance that the application did not have
a memory leak problem with PermGen space or anywhere else in the heap. I proposed that we
track heap usage after every Full GC over a period of time and see the heap usage trends over time.
The heap usage trends should provide a good indication if the application
has a problematic memory leak issue or not. </p>
<p>In order to do so, I wrote a small script to parse out only the heap and permgen space
info after a full GC from the gc log. I wrote this script in Clojure as follows:</p>
<pre class="brush: clojure;">
; Define the regex pattern to parse each line in gc log
(defn full-gc-pattern []
(let [timestamp "([\\d\\.]+): .* "
space "(\\d+)K->(\\d+)K\\((\\d+)K.+ "
new-gen space
old-gen space
perm-gen space
heap "(\\d+)K->(\\d+)K\\((\\d+)K\\)\\], "
exec-stat "(\\d+\\.\\d+).*" ]
(re-pattern (str timestamp new-gen old-gen perm-gen heap exec-stat))))
; We only want to dump timestamp, total heap and perm gen space
; after each full GC
; Variable definitions:
; ts - timestamp (in seconds)
; ys - YoungGen space starting heap size (in KB)
; ye - YoungGen space ending heap size (in KB)
; ym - YoungGen space max heap size (in KB)
; os - OldGen space starting heap size (in KB)
; oe - OldGen space ending heap size (in KB)
; om - OldGen space max heap size (in KB)
; hs - Total heap space starting heap size (in KB)
; he - Total heap space ending heap size (in KB)
; hm - Total heap space max heap size (in KB)
; ps - PermGen space starting heap size (in KB)
; pe - PermGen space ending heap size (in KB)
; pm - PermGen space max heap size (in KB)
(defn process-full-gc [entry]
(let [parsed (first (re-seq (full-gc-pattern) entry))
[a ts ys ye ym os oe om hs he hm ps pe pm & e] parsed]
(println (format "%s,%s,%s" ts he pe))))
; Load the gc log data
(def gcdata (line-seq (clojure.java.io/reader (clojure.java.io/file "gc.log"))))
; Process each full GC entry after filtering out minor GC entries
(doseq [line (keep #(re-find #".*Full GC.*" %) gcdata)]
(process-full-gc line))
</pre>
<p>Using this script against the gc log file, and taking that output and loading into a spreadsheet,
I can plot and generate a trendline to see if memory is growing or not. Here's the heap usage plot over
approximately a month.
</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrDabjqlgWyIGy-lk3n9LOUTIqYTCz6w5ngxzaXYfiABTUnErkr4uqWK1pAjJSnH0K_E6uzWAzxj8WzwrnnnuyZN4ADHHMj8bkTk_PDPpgP28FrcMb4K8Lykzuzic2hS-XQdDSgg/s1600/heaptrends.PNG" imageanchor="1" style=""><img border="0" height="274" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrDabjqlgWyIGy-lk3n9LOUTIqYTCz6w5ngxzaXYfiABTUnErkr4uqWK1pAjJSnH0K_E6uzWAzxj8WzwrnnnuyZN4ADHHMj8bkTk_PDPpgP28FrcMb4K8Lykzuzic2hS-XQdDSgg/s400/heaptrends.PNG" /></a></div>
<p>
As seen on the chart, the slope of the trendlines are small and negative, which is a good indicator
that we do not have a memory leak problem with this application.
</p>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-55139192736454860292012-05-28T06:00:00.000-07:002012-05-28T06:00:07.257-07:00Character Encoding Troubles<p>In the past, I encountered an issue with a legacy application that migrated from WebLogic Server running on a old Windows server to a newer Tomcat application server running on Red Hat Linux environment. This application has been running fine without issues for weeks until the developer suddenly started to complain that this application is not saving the registered ® and copyright © trademark symbols to the database. The developer also said that when he runs the application from his Windows laptop, he's able to save these two symbols into the database. </p>
<p>Earlier in my career, I was developer for a software company that built multilingual software specializing in Asian languages, I recognize this as a character encoding issue. So I asked the developer to send me the source code related to this issue and here's the relevant part of the code:</p>
<pre class="brush: java;">
// Sending updates to the database
update.setValue("data",encodeString(text));
// Inserting the data to the database
insert.setValue("data",encodeString(text));
</pre>
<p>That seemed odd, what does the method <code>encodeString</code> do? Here's the implementation:</p>
<pre class="brush: java;">
public String encodeString(String value) throws java.io.UnsupportedEncodingException
{
log("encodeString", "Begin");
if (value == null)
{
log("encodeString", "value == null");
return value;
}
byte [] btValue = value.getBytes();
String encodedValue = new String(btValue, _ISO88591);
/*
Charset utf8charset = Charset.forName("UTF-8");
Charset iso88591charset = Charset.forName("ISO-8859-1");
ByteBuffer inputBuffer = ByteBuffer.wrap(btValue);
// decode UTF-8
CharBuffer data = utf8charset.decode(inputBuffer);
// encode ISO-8559-1
ByteBuffer outputBuffer = iso88591charset.encode(data);
byte[] outputData = outputBuffer.array();
byte[] inputData = inputBuffer.array();
log("ISO-8859-1: ", new String(outputData));
log("UTF-8: ", new String(inputData));
//String encodedValue = new String(btValue, _ISO88591);
String encodedValue = new String(inputData);
String encodedValue_ISO88591 = new String(inputData, _ISO88591);
//encodedValue.getBytes("UTF-8");
log("Encoded UTF: ", encodedValue);
log("Encoded ISO88591: ", encodedValue_ISO88591);
*/
return encodedValue;
}
</pre>
<p>Wow! I can see that the developer is trying to get a handle on this encoding business and hence all the commented out R&D code, but clearly this developer is not familiar with character set encoding issues.</p>
<p>The main problem is with the following line of code:</p>
<pre class="brush: java;">
byte [] btValue = value.getBytes();
</pre>
<p>From Java API manual, the <code>getBytes()</code> method encodes the string into a sequence of bytes using the platform's default charset. On the old legacy Windows Server that this application was originally running on, it was probably using Windows code page 1252, which is basically ISO-8859-1 and hence the register and copyright symbols were correctly encoded. However, on the Red Hat Linux operating system, the default encoding was ascii and therefore the register/copyright symbol got converted into question marks. </p>
<p> Java strings are internally unicode (UTF-16) and typically the JDBC drivers will provide the appropriate conversions to and from the database. Therefore, fix to this application is simple, all one have to do is merely change the following two lines of code and get rid of the entire <code>encodeString()</code> method:</p>
<pre class="brush: java;">
// Changed from : update.setValue("data",encodeString(text));
update.setValue("data",text);
// Changed from : insert.setValue("data",encodeString(text));
insert.setValue("data",text);
</pre>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-64153521540289069402012-05-14T06:00:00.000-07:002012-05-14T06:00:14.804-07:00Working with Tibco EMS Message Topics using F# and Clojure<p>In my
<a href=http://jyliao.blogspot.com/2012/04/f-clojure-and-message-queues-on-tibco.html">previous blog post</a>,
I showed how to connect to a Tibco EMS Queue with F# and Clojure to represent integration interoperability
from .NET and Java platforms. Message queues are a way to implement the Request-Reply
pattern, which is one of the many enterprise integration patterns described in the book
<a href="http://www.amazon.com/exec/obidos/ASIN/0321200683/techie2biz-20">Enterprise Integration Patterns</a>.
Another basic enterprise integration pattern is the Publish-Subscribe pattern, which
can be implement via JMS topics. This blog post shows how to connect to JMS topic from .NET and Java using Tibco EMS
as the JMS provider.
</p>
<p>
Here is the F# version:
</p>
<pre class="brush: fsharp;">
#r @"C:\tibco\ems\6.3\bin\TIBCO.EMS.dll"
open System
open TIBCO.EMS
let serverUrl = "tcp://localhost:7222"
let producer = "producer"
let consumer = "consumer"
let password = "testpwd"
let topicName = "testTopic"
let subscribeToTopic serverUrl userid password topicName messageProcessor =
async {
let connection = (userid,password)
|> (new TopicConnectionFactory(serverUrl)).CreateTopicConnection
let session = connection.CreateTopicSession(false,Session.AUTO_ACKNOWLEDGE)
let topic = session.CreateTopic(topicName)
let subscriber = session.CreateSubscriber(topic)
connection.Start()
printf "Subscriber connected!\n"
while true do
try
subscriber.Receive() |> messageProcessor
with _ -> ()
connection.Close()
}
let publishTopicMessages serverUrl userid password topicName messages =
let connection = (userid,password)
|> (new TopicConnectionFactory(serverUrl)).CreateTopicConnection
let session = connection.CreateTopicSession(false,Session.AUTO_ACKNOWLEDGE)
let topic = session.CreateTopic(topicName)
let publisher = session.CreatePublisher(topic)
connection.Start()
messages
|> Seq.iter (fun item -> session.CreateTextMessage(Text=item)
|> publisher.Send)
connection.Close()
// Just dump message to console for now
let myMessageProcessor (msg:Message) =
msg.ToString() |> printf "%s\n"
let consumeMessageAsync = subscribeToTopic "tcp://localhost:7222" "consumer" "testpwd"
let produceMessages topicName messages = publishTopicMessages "tcp://localhost:7222" "producer" "testpwd" topicName messages
// Asynchronously start the topic subscriber
Async.Start(consumeMessageAsync "testTopic" myMessageProcessor)
// Publish messages to the Tibco EMS topic
[ "Aslund"; "Barrayar"; "Beta Colony"; "Cetaganda"; "Escobar"; "Komarr"; "Marilac"; "Pol"; "Sergyar"; "Vervain"]
|> produceMessages "testTopic"
printf "Done!"
</pre>
<p>One thing to point out is that Tibco, unfortunately, did not implement IDisposable for it's Connection objects;
perhaps in it's bid to stay faithful to the Java API. That design choice seems unfortunate to me in the sense that I no longer
can leverage C#'s <code>using</code> keyword or F#'s <code>use</code> keyword to automatically close connection.
I suppose it is fairly trivial to subclass the <code>QueueConnection</code> and <code>TopicConnection</code> class
and add the IDisposable interface, but I feel that
Tibco should have done this and developed the Tibco .NET API using idioms that are .NET specific.</p>
<p>Putting my rants aside, here is the equivalent Clojure code to connect to Tibco Topics:</p>
<pre class="brush: clojure;">
(import '(java.util Enumeration)
'(com.tibco.tibjms TibjmsTopicConnectionFactory)
'(javax.jms Message JMSException Session
Topic TopicConnectionFactory
TopicConnection TopicSession
TopicSubscriber))
(def serverUrl "tcp://localhost:7222")
(def producer "producer")
(def consumer "consumer")
(def password "testpwd")
(def topicName "testTopic")
;------------------------------------------------------------------------------
; Subscribe to Topic asynchronously
;------------------------------------------------------------------------------
(defn subscribe-topic [server-url user password topic-name process-message]
(future
(with-open [connection (-> (TibjmsTopicConnectionFactory. server-url)
(.createTopicConnection user password))]
(let [session (.createTopicSession connection false Session/AUTO_ACKNOWLEDGE)
topic (.createTopic session topic-name)]
(with-open [subscriber (.createSubscriber session topic)]
(.start connection)
(loop []
(process-message (.receive subscriber))
(recur)))))))
;------------------------------------------------------------------------------
; Publishing to a Topic
;------------------------------------------------------------------------------
(defn publish-to-topic [server-url user password topic-name messages]
(with-open [connection (-> (TibjmsTopicConnectionFactory. server-url)
(.createTopicConnection user password))]
(let [session (.createTopicSession connection false Session/AUTO_ACKNOWLEDGE)
topic (.createTopic session topic-name)
publisher (.createPublisher session topic)]
(.start connection)
(doseq [item messages]
(let [message (.createTextMessage session)]
(.setText message item)
(.publish publisher message))))))
; Create function aliases with connection information embedded
(defn produce-messages [topic-name messages]
(publish-to-topic "tcp://localhost:7222" "producer" "testpwd" topic-name messages))
(defn consume-messages [topic-name message-processor]
(subscribe-topic "tcp://localhost:7222" "consumer" "testpwd" topic-name message-processor))
; Just dump messages to console for now
(defn my-message-processor [message]
(println (.toString message)))
; Start subscribing messages asynchronously
(consume-messages "testTopic" my-message-processor)
; Publish to topic
(def my-messages '("alpha" "beta" "gamma" "delta"
"epsilon" "zeta" "eta" "theta"
"iota" "kappa" "lambda" "mu" "nu"
"xi", "omicron" "pi" "rho"
"signma" "tau" "upsilon" "phi",
"chi" "psi" "omega"))
(produce-messages "testTopic" my-messages)
</pre>
<p>When I fire up both scripts, the messages published to the topic would be received by both the .NET and Java clients.
With these scripts, I can easily swap out the message generators or message processors as needed for any future testing scenarios.</p>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com2tag:blogger.com,1999:blog-18281936.post-57085258424292975582012-04-30T06:00:00.000-07:002012-04-30T06:00:23.791-07:00F#, Clojure and Message Queues on Tibco EMS<body>
<script type="text/javascript" src="https://raw.github.com/sattvik/sh-clojure/master/src/shBrushClojure.js"></script>
<p>It looks like I will be getting much more hands on with Tibco EMS. Since the Tibco EMS system in use will have connections from both .NET platforms and Java platforms, I wanted to write some scripts to run some engineering tests on Tibco EMS. I decided to simulate .NET side connections with F# and Java side connections with Clojure. Taking the sample code from Tibco installation, I created the following F# script that sends messages to a queue from the sample C# code:
<pre class="brush: fsharp;">
#r @"C:\tibco\ems\6.3\bin\TIBCO.EMS.dll"
open System
open TIBCO.EMS
let serverUrl = "tcp://localhost:7222"
let producer = "producer"
let consumer = "consumer"
let password = "testpwd"
let queueName = "testQueue"
let getQueueTextMessages serverUrl userid password queueName messageProcessor =
async {
let connection = (userid,password)
|> (new QueueConnectionFactory(serverUrl)).CreateQueueConnection
let session = connection.CreateQueueSession(false,Session.AUTO_ACKNOWLEDGE)
let queue = session.CreateQueue(queueName)
let receiver = session.CreateReceiver(queue)
connection.Start()
printf "Queue connection established!"
while true do
try
receiver.Receive() |> messageProcessor
with _ -> ()
}
let sendQueueTextMessages serverUrl userid password queueName messages =
let connection = (userid,password)
|> (new QueueConnectionFactory(serverUrl)).CreateQueueConnection
let session = connection.CreateQueueSession(false,Session.AUTO_ACKNOWLEDGE)
let queue = session.CreateQueue(queueName)
let sender = session.CreateSender(queue)
connection.Start()
messages
|> Seq.iter (fun item -> session.CreateTextMessage(Text=item)
|> sender.Send)
connection.Close()
// Just dump message to console for now
let myMessageProcessor (msg:Message) =
msg.ToString() |> printf "%s\n"
let consumeMessageAsync = getQueueTextMessages "tcp://localhost:7222" "consumer" "testpwd"
let produceMessages queueName messages = sendQueueTextMessages "tcp://localhost:7222" "producer" "testpwd" queueName messages
// Start message consumer asynchronously
Async.Start(consumeMessageAsync "testQueue" myMessageProcessor)
// Send messages to the Tibco EMS
[ "Aslund"; "Barrayar"; "Beta Colony"; "Cetaganda"; "Escobar"; "Komarr"; "Marilac"; "Pol"; "Sergyar"; "Vervain"]
|> produceMessages "testQueue"
</pre>
<p>The queue consumer is implemented asynchronously so it won't block executing subsequent statements. To test Tibco JMS from Java, here is the equivalent Clojure code:</p>
<pre class="brush: clojure;">
(import '(java.util Enumeration)
'(com.tibco.tibjms TibjmsQueueConnectionFactory)
'(javax.jms Message JMSException Session
Queue QueueBrowser
QueueConnection QueueReceiver
QueueSession QueueSender))
(def serverUrl "tcp://localhost:7222")
(def producer "producer")
(def consumer "consumer")
(def password "testpwd")
(def queueName "testQueue")
; Consume Queue Text messages asynchronously
(defn get-queue-text-messages [server-url user password queue-name process-message]
(future
(with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
(.createQueueConnection user password))]
(let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE)
queue (.createQueue session queue-name)]
(with-open [receiver (.createReceiver session queue)]
(.start connection)
(loop []
(process-message (.receive receiver))
(recur)))))))
; Send multiple Text messages
(defn send-queue-text-messages [server-url user password queue-name messages]
(with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
(.createQueueConnection user password))]
(let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE)
queue (.createQueue session queue-name)
sender (.createSender session queue)]
(.start connection)
(doseq [item messages]
(let [message (.createTextMessage session)]
(.setText message item)
(.send sender message))))))
; Create function aliases with connection information embedded
(defn consume-messages [queue-name message-processor]
(get-queue-text-messages serverUrl producer password queue-name message-processor))
(defn produce-messages [queue-name messages]
(send-queue-text-messages serverUrl producer password queue-name messages))
; Just dump messages to console for now
(defn my-message-processor [message]
(println (.toString message)))
; Start consuming messages asynchronously
(consume-messages "testQueue" my-message-processor)
; Send messages to queue
(def my-messages '("alpha" "beta" "gamma" "delta"
"epsilon" "zeta" "eta" "theta"
"iota" "kappa" "lambda" "mu" "nu"
"xi", "omicron" "pi" "rho"
"signma" "tau" "upsilon" "phi",
"chi" "psi" "omega"))
(produce-messages "testQueue" my-messages)
</pre>
<p>With these scripts, I can easily swap in different message generators and message processors as needed for any testing purposes.
When I fired up both these scripts up to the part where queue consumers are running in both F# and Clojure version and then send the messages, I could see that Tibco EMS send half the messages to my F# script and the other half to my Clojure script. Since both of these scripts run in REPL environment, I can easily adjust my level of testing as I get results.</p>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com1tag:blogger.com,1999:blog-18281936.post-31269464941110702882012-04-16T06:00:00.000-07:002012-04-16T06:00:12.355-07:00F# and SharePoint 2010 Object Hierarchy/Properties<p>I had the opportunity to take a SharePoint 2010 class recently. In the class, the labs were mostly a cut and paste affair
due to time limitations. Those lab exercises only helps me to become familiar with working in Visual Studio and seeing some of
the SharePoint API, but does not really help me engage actively in thinking about what I was actually doing. After the class, I decided to
write a F# equivalent of the lab exercises to help me get a deeper understanding and to learn. The class did impress on me that the tooling
for SharePoint 2010 development is so much more superior in C# such that I've decided to build only the logic code in F# and retain the
C# SharePoint project. This blog post describes my effort in getting that first class exercise working with F# code.</p>
<p>I ended up creating an empty SharePoint 2010 C# project that references a F# library project.
I added two application pages to the SharePoint 2010 project, the first being FarmHierarchy.aspx.
In FarmHierarchy.aspx, I added the following markup between the opening and closing tags of the
<code><asp:Content></code>
element that has an <b>ID</b> of <b>Main</b>:</p>
<pre class="brush: xml;">
<h2>My Farm</h2>
<asp:TreeView ID="farmHierarchyViewer" runat="server"
ShowLines="true" EnableViewState="true"></asp:TreeView>
</pre>
<p>The second application page I created was PropertyChanger.aspx. I added the following markup between the opening and closing tags of the
<code><asp:Content></code>
elsement that has an <b>ID</b> of <b>Main</b>:</p>
<pre class="brush: xml;">
<h2>Properties:</h2>
<asp:Label ID="objectName" runat="server" Text=""></asp:Label><br/><br/>
<asp:Panel ID="webProperties" runat="server" Visible="false" BorderColor="Orange" BorderStyle="Dashed" BorderWidth="1">
<asp:Label ID="WebLabel" runat="server" Text="Web Title"></asp:Label>
<br/>
<asp:TextBox ID="webTitle" runat="server" EnableViewState="true"></asp:TextBox>
&nbsp;
<asp:Button ID="webTitleUpdate" runat="server" Text="Update"/>
&nbsp;
<asp:Button ID="webCancel" runat="server" Text="Cancel" />
</asp:Panel>
<asp:Panel ID="listProperties" runat="server" Visible="false" BorderColor="Orange" BorderStyle="Dashed" BorderWidth="1">
<asp:Label ID="ListLabel" runat="server" Text="List Properties"></asp:Label>
<br/>
<asp:CheckBox ID="listVersioning" runat="server" EnableViewState="true" Text="Enable Versioning" />
<br/>
<asp:CheckBox ID="listContentTypes" runat="server" EnableViewState="true" Text="Enable Content Types" />
&nbsp;
<asp:Button ID="listPropertiesUpdate" runat="server" Text="Update" />
&nbsp;
<asp:Button ID="listCancel" runat="server" Text="Cancel"/>
</asp:Panel>
</pre>
<p>The F# code would iterate through the services, Web applications, site collections, and lists in the SharePoint farm,
with the details added to nodes in the TreeView control:</p>
<pre class="brush: fsharp;">
module FsLab01
open System
open System.Web.UI.WebControls
open Microsoft.SharePoint
open Microsoft.SharePoint.Administration
open System.Web.UI
// Convenience methods
let nullfunc _ = ()
// Copied from Clojure
let cond clauses =
let (_,func) = clauses |> Seq.find (fun (pred,func) -> pred)
func()
let navListUrl url (id:Guid) =
sprintf "%s/_layouts/lab01/PropertyChanger.aspx?type=list&objectID=%s" url <| id.ToString()
let navWebUrl url (id:Guid) =
sprintf "%s/_layouts/lab01/PropertyChanger.aspx?type=web&objectID=%s" url <| id.ToString()
// Recursively add SPWeb & SPList objects
let rec addWeb (web:SPWeb) (parentNode:TreeNode) =
let node = new TreeNode(web.Title,null,null,navWebUrl web.Url web.ID, "_self")
node |> parentNode.ChildNodes.Add
[0..(web.Lists.Count-1)]
|> Seq.map (fun i -> web.Lists.[i])
|> Seq.iter (fun item ->
new TreeNode(item.Title,null,null,navListUrl web.Url item.ID,"_self")
|> node.ChildNodes.Add)
[0..(web.Webs.Count-1)]
|> Seq.map (fun i -> web.Webs.[i])
|> Seq.iter (fun item -> try addWeb item node finally item.Dispose())
// Main function to be called by FarmHierarchy.aspx code to build TreeView control
let loadViewer (viewer:TreeView) (farm:SPFarm) =
let processSite (webappnode:TreeNode) (site:SPSite) =
site.CatchAccessDeniedException <- false
try
let node = new TreeNode(Text=site.Url)
node |> webappnode.ChildNodes.Add
addWeb site.RootWeb node
finally
site.CatchAccessDeniedException <- false
let processWebApp (svcNode:TreeNode) (webapp:SPWebApplication) =
let node = new TreeNode(Text=webapp.DisplayName )
node |> svcNode.ChildNodes.Add
cond [(not webapp.IsAdministrationWebApplication,
fun _ -> webapp.Sites |> Seq.iter (processSite node));
(true,nullfunc)]
let processService (svc:SPService) =
let label = sprintf "FarmService (Type=%s; Status=%s)" (svc.TypeName) (svc.Status.ToString())
let node = new TreeNode(Text=label)
node |> viewer.Nodes.Add
match svc with
| :? SPWebService as websvc -> websvc.WebApplications |> Seq.iter (processWebApp node)
| _ -> ()
viewer.Nodes.Clear()
farm.Services |> Seq.iter processService
viewer.ExpandAll()
</pre>
<p>As I was creating <code>loadViewer</code> method, I was bothered by the if-else-then clauses in the code.
I had a previous <a href="http://jyliao.blogspot.com/2012/01/translating-if-then-else-control-flow.html">blog</a> that talked about this issue when it struck me that what I really wanted
was something similar to the <code>cond</code> macro in Clojure. Hence the convenience function called <code>cond</code>
in the above F# code.</p>
<p>With the F# code written and packaged as a library, I can now use the above F# function in FarmHierarchy.aspx as follows:</p>
<pre class="brush: csharp;">
protected void Page_Load(object sender, EventArgs e)
{
SPFarm thisFarm = SPFarm.Local;
FsLab01.loadViewer(farmHierarchyViewer, thisFarm);
}
</pre>
<p>The second part of this lab was to create code to manipulate properties of SharePoint <code>SPWeb</code> or <code>SPList</code>
objects. The F# code that manipulates the SharePoint object properties and the UI are as follow:</p>
<pre class="brush: fsharp;">
let changeProperty (page:Page) (objectName:Label) (webtitle:TextBox)
(listpanel:Panel) (webpanel:Panel)
(listVersioning:CheckBox) (listContentTypes:CheckBox)
(webUpdateBtn:Button) (listUpdateBtn:Button)
(webCancelBtn:Button) (listCancelBtn:Button) =
let homeurl baseurl = sprintf "%s/_layouts/lab01/FarmHierarchy.aspx" baseurl
let wrapupdates (web:SPWeb) action =
web.AllowUnsafeUpdates <- true
action()
web.AllowUnsafeUpdates <- false
let hidepanels () =
listpanel.Visible <- false
webpanel.Visible <- false
let cancel (web:SPWeb) =
page.Response.Redirect(homeurl web.Url)
let checkNull item =
cond [(item=null, fun _ -> objectName.Text <- "Malformed URL"
hidepanels()
"");
(item<>null, fun _ -> item.ToString())]
try
let objectType = checkNull page.Request.["type"]
let objectId = checkNull page.Request.["objectID"]
match objectType with
| "web" ->
listpanel.Visible <- false
webpanel.Visible <- true
use web = SPContext.Current.Site.OpenWeb(new Guid(objectId))
// Hook up the events
let myupdates _ =
web.Title <- webtitle.Text
web.Update()
webUpdateBtn.Click.Add(
fun _ ->
try
wrapupdates web myupdates
page.Response.Redirect(homeurl web.Url)
with ex ->
objectName.Text <- ex.Message
hidepanels()
)
webCancelBtn.Click.Add(fun _ -> cancel web)
objectName.Text <- sprintf "Web: %s" web.Title
cond [(not page.IsPostBack,
fun () -> webtitle.Text <- web.Title);
(true,nullfunc)]
| "list" ->
listpanel.Visible <- true
webpanel.Visible <- false
let web = SPContext.Current.Web
let splist = web.Lists.[new Guid(objectId)]
let myupdates _ =
splist.EnableVersioning <- listVersioning.Checked
splist.ContentTypesEnabled <- listContentTypes.Checked
splist.Update()
listUpdateBtn.Click.Add(
fun _ ->
try
wrapupdates web myupdates
page.Response.Redirect(homeurl web.Url)
with ex ->
objectName.Text <- ex.Message
hidepanels())
listCancelBtn.Click.Add(fun _ -> cancel web)
cond [(not page.IsPostBack,
fun _ -> listVersioning.Checked <- splist.EnableVersioning
listContentTypes.Checked <- splist.ContentTypesEnabled)]
| _ -> ()
with ex-> objectName.Text <- ex.Message
</pre>
<p>I needed to sign my F# code in order it to work in SharePoint. Since the C# SharePoint 2010 already created the key, all I had to do is to
add <code>--keyfile:<Location to keyfile>\key.snk</code> to the F# build setting's "Other Flags" properties.
In addition, before I can build and deploy, I needed to add the F# library dll to the C# SharePoint project's package.
I can do so by clicking on Package in the SharePoint project and then click on the Advanced button.</p>
<p>I can now call this F# function from PropertyChanger.aspx code as follows:</p>
<pre class="brush: csharp;">
FsLab01.changeProperty(this.Page, objectName, webTitle,
listProperties, webProperties,
listVersioning, listContentTypes,
webTitleUpdate, listPropertiesUpdate,
webCancel,listCancel);
</pre>
<p>Here's how what FarmHierarchy.aspx would look like :</p>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjL24M3lr9poKWknqlYRnes26hrLFfeRpgY_mV56IyMA_R6PJlNSdzrVHOjfB8yxiXm8Y9yR235KAT4vQGcXYW0vWXQ7YPPjCv00qK_KMANlxJZ2Dw8OcyqBhVlYMDOmuD9SL1fKQ/s1600/farmhierarchy.png" imageanchor="1" style=""><img border="0" height="228" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjL24M3lr9poKWknqlYRnes26hrLFfeRpgY_mV56IyMA_R6PJlNSdzrVHOjfB8yxiXm8Y9yR235KAT4vQGcXYW0vWXQ7YPPjCv00qK_KMANlxJZ2Dw8OcyqBhVlYMDOmuD9SL1fKQ/s400/farmhierarchy.png" /></a></div>
<p>Here's how what PropertyChanger.aspx would look like if I wanted to modify a SPWeb object:</p>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS4v6tJyTmhktRii5gJMhqs3_GoNyipq3LFp4BS4qu5z96g63gLwnooy8Bc7qL3gaqCD2XQ30gliWfTVccEjU1PY44GR4_Nn-_yOdgAiNFMa2zSAHWsfNUCdE6cko69cxmTn-XdA/s1600/ModifySPWeb.png" imageanchor="1" style=""><img border="0" height="226" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS4v6tJyTmhktRii5gJMhqs3_GoNyipq3LFp4BS4qu5z96g63gLwnooy8Bc7qL3gaqCD2XQ30gliWfTVccEjU1PY44GR4_Nn-_yOdgAiNFMa2zSAHWsfNUCdE6cko69cxmTn-XdA/s400/ModifySPWeb.png" /></a></div>
<p>Here's how what PropertyChanger.aspx would look like if I wanted to modify a SPList object:</p>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1EdfqutNpjg0CNIoxgVjziMtpoH2Cf_l9Uj9Tsmy4YQ0oSPdCFcJ6D2HRA9HrboSb6NMk_ZzK6VI5vRXmtvB_3MFNlqexjrKRa6Qwkdix2mirZHkeGwKPs2KlwqdTsDFE3BI6xQ/s1600/ModifySPList.png" imageanchor="1" style=""><img border="0" height="229" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1EdfqutNpjg0CNIoxgVjziMtpoH2Cf_l9Uj9Tsmy4YQ0oSPdCFcJ6D2HRA9HrboSb6NMk_ZzK6VI5vRXmtvB_3MFNlqexjrKRa6Qwkdix2mirZHkeGwKPs2KlwqdTsDFE3BI6xQ/s400/ModifySPList.png" /></a></div>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com1tag:blogger.com,1999:blog-18281936.post-23860069874303362132012-03-26T06:00:00.000-07:002012-03-26T06:00:02.557-07:00F# and Windows Server AppFabric Cache<p>I recently started investigating Windows Server AppFabric, which is different from Windows Azure AppFabric, and trying understand how to use AppFabric, what are some of the operational support implications and functional capabilities and limitations. To help me get going, I have been reading the book <a href="http://www.amazon.com/gp/product/1430228172?tag=techie2biz-20">Pro Windows Server AppFabric</a> by Stephen Kaufman and going through the <a href="http://www.microsoft.com/download/en/details.aspx?id=7956"> Windows Server AppFabric Training Kit</a>. The free downloadable training kit was more valuable in helping me understand Windows Server AppFabric.</p>
<p>As always, the only way to truly learn new technology is to take it out for a spin. Here are some of the powershell scripts that I've used to create the cache and get it going:</p>
<pre class="brush: powershell;">
# Import the necessary administration modules
Import-Module DistributedCacheAdministration
Import-Module DistributedCacheConfiguration
# List the available powershell commands
Get-Command -module DistributedCacheAdministration
Get-Command -module DistributedCacheConfiguration
# These were run on the localhost of AppFabric
Use-CacheCluster
# Start the cache cluster
Start-CacheCluster
# Check to make sure it is up and running
Get-CacheHost
# Create a new cache
New-Cache -CacheName MyTestCache -TimeToLive 60 -Expirable true
# Check cache is created
Get-Cache
# Grant local user access to cache
Grant-CacheAllowedClientAccount MyUserId
# Check security settings
Get-CacheAllowedClientAccounts
# After a few runs, check Cache statistics
Get-CacheStatistics -CacheName MyTestCache
# Get Cache configuration information
Get-CacheConfig MyTestCache
# Creating a Cache with HA - all server needs to be on
# Windows Server Enterprise Edition
# Any host not on Enterprise Edition will not start cache cluster
New-Cache MyNewHACache -Secondaries 1
# Modifying the cache to enable callbacks
Set-CacheConfig -CacheName MyTestCache -NotificationsEnabled true -TimeToLive 180
</pre>
<p>For the client, I wrote some F# scripts to experiment with Windows Server AppFabric. I ran these on the local server and the loopback adapter enabled:</p>
<pre class="brush: fsharp;">
// I copied the libraries from c:\windows\system32\appfabric
#r @"C:\lib\Microsoft.ApplicationServer.Caching.Client.dll"
#r @"C:\lib\Microsoft.ApplicationServer.Caching.Core.dll"
open System
open System.Collections.Generic
open Microsoft.ApplicationServer.Caching
// Make sure you grant
// grant-cacheallowedclientaccount my-user-id
// Expensive operation, do this once on startup
let dcf = new DataCacheFactory(
let servers = new List<DataCacheServerEndpoint>()
new DataCacheServerEndpoint("localhost",22233) |> servers.Add
new DataCacheFactoryConfiguration(Servers=servers))
printfn "Data Cache Factory Created!"
// Testing add/get in default region
let cache = dcf.GetCache("MyTestCache")
let retval = cache.Add("mykey","hello app fabric!")
cache.Get("mykey")
// Create a new region in the cache, this will pin to a single cache node
cache.CreateRegion("stocks")
// Helper function to create DataCacheTags
let createTags tags = Seq.map (fun tag -> new DataCacheTag(tag)) tags
type Company =
{ Symbol : string; Name: string; Address : string; Phone : string; Tags :seq<string> }
// Create a bunch of company info and put it in the "stocks" region
[{Symbol="AAPL"; Name="Appl Inc."; Phone="408-996-1010";
Address="1 Infinite Loop, Cupertino, CA 95014";
Tags = ["Technology";"Nasdaq";"Personal Computers"]};
{Symbol="CAT"; Name="Caterpillar, Inc."; Phone="309-675-1000";
Address="100 North East Adams Street, Peoria, IL 61629";
Tags = ["Dow";"Industrial Goods";"Farm & Construction Machinery"]};
{Symbol="ACI"; Name="Arch Coal, Inc."; Phone="314-994-2700";
Address="Once City Place Drive, Suite 300, St. Louis, MO 63141";
Tags = ["Basic Materials";"Industrial Metals & Materials"]};
{Symbol="HP"; Name="Hewlett-Packard Company"; Phone="650-857-1501";
Address="3000 Hanover Street, Palo Alto, Ca 94304";
Tags = ["Dow";"Technology"; "Diversified Computer Systems"]};
{Symbol="JPM"; Name="JP Morgan Chase & Co."; Phone="212-270-6000";
Address="270 Park Avenue, New York, NY 10017";
Tags = ["Dow";"Financial"; "Money Center Banks"]};
{Symbol="XOM"; Name="Exxon Mobile Corporation"; Phone="972-444-1000";
Address="5959 Las Colinas Boulevard, Irving, TX 75039-2298";
Tags = ["Dow";"Basic Materials"; "Major Integrated Oil & Gas"]};]
|> Seq.iter (fun item -> cache.Put(item.Symbol, item, (createTags item.Tags),"stocks") |> ignore)
// Get all stocks in "stocks" region as a HashMap
let stocks = cache.GetObjectsInRegion("stocks")
// Bulk get, only available with defined regions not default region
let myportfolio = ["AAPL";"CAT";"HP";"JPM"]
let mystocks = cache.BulkGet(myportfolio,"stocks")
// Getting objects from cache by tags (again, only available in defined region)
cache.GetObjectsByTag(new DataCacheTag("Dow Jones"),"stocks")
// Getting stuff by all tags (AND filter)
let candidateTags = ["Technology"; "Personal Computers"] |> createTags
cache.GetObjectsByAllTags(candidateTags,"stocks")
// Getting stuff by any tags (OR filter)
let interestTags = ["Technology"; "Financial"; "Basic Materials"] |> createTags
cache.GetObjectsByAnyTag(interestTags,"stocks")
// Concurrency policy is strictly done by the client
// No explicit cache server control on concurrency
// Optimistic Currency example
cache.Remove("mykey")
let version = cache.Add("mykey","mystuff")
// Pretend another thread snuck in and modified this cache item.
cache.Put("mykey","changed value")
// This will check that somebody else modified this and throw an exception
cache.Put("mykey","another value",version)
// Pessimistic locking - locks across all cache nodes
let mutable lockHandle:DataCacheLockHandle = null
let item = cache.GetAndLock("mykey",TimeSpan.FromMinutes(60.0), &lockHandle,true)
// This will throw error due to the first lockHandle
let mutable lockHandle2:DataCacheLockHandle = null
// This will throw an exception as it was already previously locked
let item2 = cache.GetAndLock("mykey",TimeSpan.FromMinutes(60.0),&lockHandle2, true)
// This will totally ignore locking policies and overwrite things
// Locking is controlled by diligence on the client side.
cache.Put("mykey","no enforced concurrency on the server")
// Finally, this will update the data and unlock this item in the cache
// Need to redo the GetAndLock for this to work. Previous statement would wipe out the lock
cache.PutAndUnlock("mykey","new stuff",lockHandle)
// Working with Callbacks
(DataCacheOperations.AddItem,
fun cacheName regionName key version cacheOperation nd ->
printfn "Item added to cache : %s\n" key)
|> cache.AddCacheLevelCallback
|> ignore
// When this is call, it could be seconds or minutes before anything
// is printed to the console
cache.Add("newkey","new stuff")
</pre>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-33130719127989635332012-03-12T06:00:00.000-07:002012-03-12T06:00:11.533-07:00F# and SharePoint 2010<style>
.red { color: #900; }
</style>
<p>I bought a bunch of SharePoint 2010 books and haven't had the time to go through those books. Recently, I finally had some time to take look at those books and try it out in F#. Trying out examples with SharePoint 2007 was fairly painless as I get to do everything in a 32-bit environment. Trying to get Visual Studio 2010 F# to work with SharePoint 2010 was more challenging. While I was able to get the compiled F# code to work, I couldn't overcome the hurdles to get SharePoint 2010 to work with F# Interactive, which is really my prefer way to investigate SharePoint server object models. I can tell from this
<a href="http://stackoverflow.com/questions/7597426/f-script-sharepoint-2010-api-doesnt-work">Stack Overflow entry</a>
that I was not alone in having trouble to get F# interactive to work with SharePoint API. The combination of .NET 3.5 framework and 64-bit defeated my effort at getting it to work. I have tried
<a href="http://ig2600.blogspot.com/2010/05/making-fsharp-interpreter-fsi-run-in.html">Igor Dvorkin's steps to run F# interactive in 64-bit</a>
and then taking the command line arguments for the F# compiler and apply to F# interactive where applicable. So I would start up my F# interactive with the following command line parameters:</p>
<pre>
"C:\Program Files (x86)\Microsoft F#\v4.0\fsi.exe" ^
--debug:full ^
--noframework ^
--define:DEBUG ^
--define:TRACE ^
--optimize- ^
--tailcalls- ^
-r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\FSharp\2.0\Runtime\v2.0\FSharp.Core.dll" ^
-r:"C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\ISAPI\Microsoft.SharePoint.dll" ^
-r:C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll ^
-r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll" ^
-r:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.dll ^
-r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.0\System.ServiceModel.dll"
</pre>
<p>When I try to run with the above arguments, I get the following error:</p>
<pre class="red">
error FS0084: Assembly reference
'C:\Program Files (x86)\Reference Assemblies\Microsoft\FSharp\2.0\Runtime\v2.0\FSharp.Compiler.Interactive.Settings.dll'
was not found or is invalid
</pre>
<p>So something in <code>FSharp.Compiler.Interactive.Settings.dll</code> is not working with .NET Framework 3.5. I wish there was an option in F# interactive that allows me to target the runtime .NET framework to use (e.g. --target:3.5)</p>
<p>The good news is that the F# compiler works fine as long as you make sure that you are targeting .NET 3.5 runtime and x64 CPU. Here's the F# version of a sample code that I pulled from the book
<a href="http://www.amazon.com/exec/obidos/ASIN/0735627460/techie2biz-20">Inside Microsoft SharePoint 2010</a> by Ted Pattison, Andrew Connell, Scott Hillier and David Mann</p>
<pre class="brush: fsharp;">
open System
open Microsoft.SharePoint
let siteUrl = "http://localhost/"
let sites = new SPSite(siteUrl)
let site = sites.RootWeb
http://www.blogger.com/blogger.g?blogID=18281936#editor/target=post;postID=3313071912798963533
seq {for i in 0..(site.Lists.Count-1) do
if site.Lists.[i].Hidden <> true then
yield (site.Lists.[i]) }
|> Seq.iter (fun item -> printf "%s\n" item.Title)
</pre>
<p>For those who have read my past posts on F# and SharePoint and my past blog post on <a href="http://jyliao.blogspot.com/2011/11/revisiting-sharepoint-collection.html">Revisiting the SharePoint collection adapter for F#</a>, I am sorry to say that I have been led astray by C# example codes and my brain temporary malfunctioned as it produced the SharePoint adapter nonsense in that previous blog post. There was no need to write any SharePoint utility library in C# to get F# to work with SPListCollection as sequences. In the above example, I used sequence expressions to generate SPList sequences from SPListCollection. Another way is to simply create a sequence by use map function as shown in the following example:</p>
<pre class="brush: fsharp;">
// Create a sequence by mapping over all items in the collection
let spcollection = site.Lists
[0..spcollection.Count-1]
|> Seq.map (fun i -> spcollection.[i])
|> Seq.iter (fun item -> printf "%s\n" item.Title)
</pre>
<p>There was no need to leave the confines of F# development and it is also type safe compared to my previous clumsy attempt at converting <code>SPListCollection</code> into a sequence. If you really do want to use an adapter, it could also be implemented in F# as follows:</p>
<pre class="brush: fsharp;">
// SPCollectionAdapter written in F#
let fromSPCollection<'a,'b when 'b :> SPBaseCollection> (collection:'b) =
let enumerator = collection.GetEnumerator()
let rec enumerate (e:Collections.IEnumerator) acc =
let flag = e.MoveNext()
if flag = false then
acc
else
enumerate e (e.Current :?> 'a :: acc)
enumerate enumerator []
// Example usages
fromSPCollection<SPList,SPListCollection> site.Lists
|> Seq.iter (fun item -> printf "%s\n" item.Title)
// webapp is an instance of SPWebApplication object
fromSPCollection<SPContentDatabase,SPContentDatabaseCollection> webapp.ContentDatabases
|> Seq.iter (fun db ->
printf "Content Database : %s\n" db.Name
printf "Connection String : %s\n" db.DatabaseConnectionString)
</pre>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-24453112224554690732012-03-06T06:00:00.000-08:002012-04-05T14:43:38.206-07:00Adventures in troubleshooting out of memory errors with Coherence cluster.<script type="text/javascript" src="https://raw.github.com/sattvik/sh-clojure/master/src/shBrushClojure.js"></script>
<p>One day, an application team manager called me and said that their application caused an out of memory error condition in their Oracle Coherence cluster. This same code base ran in the old Coherence 3.1 environment for months without running into out of memory conditions and now is failing in the new Coherence 3.6 environment in matter of a few weeks on a regular basis. He said that he had heap dumps and logs and asked whether I could take a look at it and troubleshoot it.</p><p>Initially, I was skeptical about being able to help this team manager out. After all, I know almost nothing about their application code and in all practical terms, I had no previous development experience with Coherence with the exception that I read the book <a href="http://www.amazon.com/gp/product/1847196128?tag=techie2biz-20">Oracle Coherence 3.5</a> by Aleksandar Seovic in the past. My previously participated in testing Coherence performance on VMware and that really did not require me to delve into the Coherence API at all.
</p>
<p>
Despite these misgivings, I decided to provide my support and told the application team manager that I'll try my best.</p>
<p>
The system with problems was a multi-node Coherence cluster. When I took a look at the logs, all of them had these similar verbose GC output:</p>
<pre>
[GC [1 CMS-initial-mark: 1966076K(1966080K)] 2083794K(2084096K), 0.1923110 secs] [Times: user=0.18 sys=0.00, real=0.19 secs]
[Full GC [CMS[CMS-concurrent-mark: 1.624/1.626 secs] [Times: user=3.22 sys=0.00, real=1.62 secs]
(concurrent mode failure): 1966079K->1966078K(1966080K), 6.6177340 secs] 2084093K->2084082K(2084096K), [CMS Perm : 13617K->13617K(23612K)], 6.6177900 secs] [Times: user=8.21 sys=0.00, real=6.62 secs]
[Full GC [CMS: 1966078K->1966078K(1966080K), 4.1110330 secs] 2084093K->2084089K(2084096K), [CMS Perm : 13617K->13615K(23612K)], 4.1111070 secs] [Times: user=4.11 sys=0.00, real=4.11 secs]
[Full GC [CMS: 1966078K->1966078K(1966080K), 4.2973090 secs] 2084092K->2084087K(2084096K), [CMS Perm : 13615K->13615K(23612K)], 4.2973630 secs] [Times: user=4.28 sys=0.00, real=4.30 secs]
[Full GC [CMS: 1966078K->1966078K(1966080K), 4.1831450 secs] 2084093K->2084093K(2084096K), [CMS Perm : 13615K->13615K(23612K)], 4.1831970 secs] [Times: user=4.18 sys=0.00, real=4.18 secs]
[Full GC [CMS: 1966078K->1966078K(1966080K), 4.2524850 secs] 2084093K->2084093K(2084096K), [CMS Perm : 13615K->13615K(23612K)], 4.2525380 secs] [Times: user=4.24 sys=0.00, real=4.25 secs]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid23607.hprof ...
Heap dump file created [2274434953 bytes in 28.968 secs]
</pre>
<p>This garbage collection log output tells me that they are using CMS for the JVM GC. The concurrent mode failure entries certainly grabbed my attention. Normally one would fix concurrent mode failures by tuning the CMS initiation occupancy fraction via -XX:CMSInitiatingOccupancyFraction flag, but in this case, looking that the heap numbers in the lines labeled "Full GC" showed that GC could not clean up any memory at all. So this problem could not be solved by GC tuning. By the way, for a great book on tuning garbage collection, I would recommend Charlie Hunt's book <a href="http://www.amazon.com/gp/product/0137142528?tag=techie2biz-20">Java Performance</a>.</p>
<p>My next step was to take a look a the heap. The heap was slightly over 2 GB, which was expected since Coherence cluster node was each configured with a 2GB heap. Well, that presented a problem for me because I'm still mainly working on a 32-bit Windows laptop. I needed to find a 64-bit system with preferably 4 GB of ram or more to look at this. Once I was able to get such a machine and fired up <a href="http://www.eclipse.org/mat/">Eclipse Memory Analyzer Tool (MAT)</a>. Once I looked at the heap, it was pretty obvious what was the biggest memory offender. The biggest memory offender was a top level hashmap chewing up 1.6 GB of memory. Delving further into that hash map structure, it reveals that Coherence caching structure is a hash of hashes. Looking at the hashes, I notice that there were over 2000+ items in the top level hash. That would imply that there were over 2000+ caches in the Coherence cluster. Studying each individual cache, I would notice cache names like</p>
<ul>
<li>alpha-FEB-21</li>
<li>alpha-FEB-22</li>
<li>alpha-FEB-23</li>
<li>alpha-FEB-24</li>
<li>beta-FEB-21</li>
<li>beta-FEB-22</li>
<li>beta-FEB-23</li>
<li>beta-FEB-24</li>
</ul>
<p>and so forth. I ask the application team manager whether he expected
to have this many caches in the cluster. The application team manager said no; he expected a much smaller set of caches. The application normally destroy caches older than 2 days. The developers provided me their code related to the creation and destruction of caches and I saw the following lines of code and it seems pretty innocuous:</p>
<pre class="brush: java;">
public static void destroyCache(String name) {
Collection listOfCacheNames = getListOfCacheNames(name, false);
Iterator iterator = listOfCacheNames.iterator();
while (iterator.hasNext()) {
String name = (String) iterator.next();
NamedCache namedCache = CacheFactory.getCache(name);
namedCache.destroy();
}
}
</pre>
<p>
I went back to the memory analyzer tool and performed a GC to root analysis and saw the top level object that's holding onto this heap as:</p>
<pre>
com.tangosol.coherence.component.net.Cluster$IpMonitor @ 0x77ff4b18 </pre>
<p>with the label "Busy Monitor" next to it. This line item seems to suggest that there's a monitor lock on this cache. Looking at the Coherence API documentation, I see the following entry:</p>
<hr/>
<h3>destroy</h3>
<pre class="oac_no_warn">
void <b>destroy</b>()
</pre>
<dl>
<dd>Release and destroy this instance of NamedCache.
<p><b>Warning:</b> This method is used to completely destroy the specified cache across the cluster. All references in the entire cluster to this cache will be invalidated, the cached data will be cleared, and all internal resources will be released.</p>
<p>Caches should be destroyed by the same mechansim in which they were obtained. For example:</p>
<ul>
<li>new Cache() - cache.destroy()</li>
<li>CacheFactory.getCache() - CacheFactory.destroyCache()</li>
<li>ConfigurableCacheFactory.ensureCache() - ConfigurableCacheFactory.destroyCache()</li>
</ul>
Except for the case where the application code expicitly allocated the cache, this method should not be called by application code.</dd>
</dl>
<hr/>
<p>
Looking at this documentation, we initially thought that since the cache was obtained via CacheFactory and therefore should be destroyed via CacheFactory ergo CacheFactory had a monitor lock on the underlying collections. The code provided by the developers used one mechanism to create the cache and another mechanism to destroy the cache so we presume that was the problem. So I implemented a test script to test out that theory and surprisingly, even destroying via CacheFactory, I still encounter out of memory issues. Only by clearing the cache before destroying the cache was I able to avoid out of memory errors. Here's the script that I developed in Clojure to test my theories:</p>
<pre class="brush: clojure;">
(import '(org.apache.commons.lang3 RandomStringUtils)
'(java.math BigInteger)
'(java.util Random Date HashMap)
'(com.tangosol.net NamedCache CacheFactory CacheService Cluster))
(defn random-text [] (RandomStringUtils/randomAlphanumeric 1048576))
(defn random-key [] (RandomStringUtils/randomAlphanumeric 12))
(CacheFactory/ensureCluster)
(def buffer (new HashMap))
(defn print-horizontal-line [c] (println (apply str (repeat 80 c))))
(def caches '("alpha" "beta" "gamma" "delta"
"epsilon" "zeta" "eta" "theta"
"iota" "kappa" "lambda" "mu" "nu"
"xi", "omicron" "pi" "rho"
"signma" "tau" "upsilon" "phi",
"chi" "psi" "omega"))
(defn load-cache [cache-name n]
(let [cache (CacheFactory/getCache cache-name)]
(print-horizontal-line "=")
(println "Creating cache : " cache-name)
(print-horizontal-line "=")
(.clear buffer)
(dotimes [_ n] (.put buffer (random-key) (random-text)))
(.putAll cache buffer)))
(defn recreate-oom-problem [cache-name]
(let [cache (CacheFactory/getCache cache-name)]
(load-cache cache-name 200)
(.destroy cache)))
(defn try-fix-oom-1 [cache-name]
(let [cache (CacheFactory/getCache cache-name)]
(load-cache cache-name 200)
(CacheFactory/destroyCache cache)))
(defn try-fix-oom-2 [cache-name]
(let [cache (CacheFactory/getCache cache-name)]
(load-cache cache-name 200)
(.clear cache)
(CacheFactory/destroyCache cache)))
; Test run recreation of original problem. Was able to reproduce OOM issues ;
(doseq [cache caches] (recreate-oom-problem cache))
; Surprise! Still have OOM issues
(doseq [cache caches] (try-fix-oom-1 cache))
; No longer have OOM issues, but memory is still leaking (slowly)
(doseq [cache caches] (try-fix-oom-2 cache))
</pre>
<p>However, I still suspect memory leaks, it's just that my memory leak is a lot smaller now. To verify that I had a memory leak, I would run my Clojure test script and the deliberately create and fill a cache without clearing it. I then forced a full garbage collection followed by a heap dump. In memory analyzer tool, I would look up the cache that I did not clear, and list all the incoming references. Then I would look for a HashMap in the incoming references and select one of those and check for outgoing references. And in that outgoing references, I could see that the key contains the name of a cache that I had called <code>CacheFactory.destroyCache()</code> on and the retained heap sizes range anywhere from 24 to 160 with the sizes that seems proportional to the size of the cache name.</p><p>In conclusion, it would seem Oracle Coherence does have a memory leak issues with the cache creation and destruction process. If we clear the cache before destroying the cache, I suspect it would be a long time before the memory leak is even noticeable by this particular application.</p>
<p>To verify that this leak did not exist in the older 3.1 version, we ran this test code on and and was unable to reproduce the out of memory errors. We also tested this against Oracle Coherence 3.7.1 and was unable to reproduce the out of memory error. So, it looks like that this memory error is specific to Oracle Coherence 3.6 only.</p>
<p>Throughout this entire process, I thought that the secret sauce that enabled me to quickly learn Coherence, reproduce and troubleshoot the Coherence out of memory problem was Clojure. Clojure allowed me to interactively manipulate Coherence clusters and explore the API, which would have been a lot slower if I had to go through the normal edit-compile-run cycle with plain old Java.</p>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com2tag:blogger.com,1999:blog-18281936.post-5223940478957723642012-02-21T06:00:00.000-08:002012-02-21T06:00:06.369-08:00Coding for reuse versus coding for specific use<p>One day, a C# developer came to me asked me to troubleshoot their application, which involves some data transformation code. In the course of troubleshooting this particular application's runtime issue, I saw this particular code implementation:</p>
<pre class="brush: csharp;">
private string[] SplitTextwithSpecialCharacter(string lineFromGUI, int splitLength)
{
List<string> lines = new List<string>();
int start = 0;
while (start < lineFromGUI.Length)
{
string temp = string.Empty;
//if length is longer it will throw exception so size the last segment appropriately
string line = lineFromGUI.Substring(start, Math.Min(splitLength, lineFromGUI.Length - start));
//temp = line.Replace(SpecialCharacter, string.Empty);
temp = line.Replace("^", ""); //New 6/30
if (temp.Length > 0)
{
//lines.Add(line.Replace(SpecialCharacter, string.Empty));
lines.Add(temp);
}
//lines.Add(line);
start += splitLength;
}
return lines.ToArray();
}
</pre>
<p>When I saw this code, I inwardly cringed due to the very narrow scope in the code implementation to solve a seemingly generic problem. This code screams for refactoring to me. From the commented out section of the code, I could see that one of the developers tried to implement some reuse before reverting back to the specific use.</p>
<p>I see a couple of issues with this particular implementation. The first issue being that this method is trying to do two very different things; one is trying to split the text into lines and the second is removing all the caret characters from the text. In my opinion, these 2 different operations should be separated out into 2 different methods. The second issue is that the caret character removal function should be generalized such that you should be able to remove any character or perhaps even a set of characters beyond just the caret symbol. In the specific implementation, what would happen if they needed code to remove an asterisk instead of caret? What would happen if they need to remove more than one special character? Do they add additional methods? Do they modify the existing method? If they modify the existing method to take parameters, then they would have to change all the existing client code that calls this method. So it would seem to be easier for future maintenance to go ahead and make these 2 functions reusable.</p><p>Here's how I would refactor this particular method into some utilities namespace and use extension methods to add capability to the string object. Here's how that code would look like:</p>
<pre class="brush: csharp;">
public static class Extensions
{
public static string FilterOut(this string line, ISet<char> filter)
{
if (line == null || filter == null || filter.Count == 0) return line;
return new String(line.Where(c => !filter.Contains(c)).ToArray());
}
public static IList<string> SplitIntoLines(this string data, int splitLength)
{
// Check preconditions
if (data == null) throw new NullReferenceException();
if (splitLength <= 0) throw new IndexOutOfRangeException("Must be greater than 0!");
IList<string> lines = new List<string>();
while (data.Length > splitLength)
{
string line = data.Substring(0,splitLength);
data = data.Substring(splitLength, data.Length-splitLength);
lines.Add(line);
}
lines.Add(data);
return lines;
}
}
</pre><p>With this extension method, you can now use this utility method as follows:</p>
<pre class="brush: csharp;">
int linesize = 30;
ISet<char> filter = new HashSet<char> { '^', '*', '&', '$' };
string[] results = data.SplitIntoLines(linesize)
.Select(line => line.FilterOut(filter))
.ToArray();
</pre>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-45591584576604669232012-02-04T07:43:00.000-08:002012-02-04T07:43:36.995-08:00Detect multiple occurrences of Java classes in jar files<style>
#mytable {
width: 700px;
padding: 0;
margin: 0;
}
caption {
padding: 0 0 5px 0;
width: 700px;
font: italic 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
text-align: right;
}
th {
font: bold 11px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #4f6b72;
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
border-top: 1px solid #C1DAD7;
letter-spacing: 2px;
text-transform: uppercase;
text-align: left;
padding: 6px 6px 6px 12px;
background: #CAE8EA url(images/bg_header.jpg) no-repeat;
}
th.nobg {
border-top: 0;
border-left: 0;
border-right: 1px solid #C1DAD7;
background: none;
}
td {
border-right: 1px solid #C1DAD7;
border-bottom: 1px solid #C1DAD7;
background: #fff;
padding: 6px 6px 6px 12px;
color: #4f6b72;
}
td.alt {
background: #F5FAFA;
color: #797268;
}
th.spec {
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #fff url(images/bullet1.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
th.specalt {http://www.blogger.com/post-edit.g?blogID=18281936&postID=4559158457660466923
border-left: 1px solid #C1DAD7;
border-top: 0;
background: #f5fafa url(images/bullet2.gif) no-repeat;
font: bold 10px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
color: #797268;
}
</style>
<p>A developer came to me for help and said that a recent upgrade to a newer Java library broke the code. The developer was getting was a class not found exception. The developer said that the old jar file was replaced with a newer jar file. I know this same library upgrade in other applications did not have any problems, so this sounded like an issue with duplicate occurrence of the same class in the classpath. Since my <a href="http://www.amazon.com/exec/obidos/ASIN/0596516177/techie2biz-20">Ruby Programming Language</a> book was in handy access when this developer came to me, I ended up writing a small ruby script that went through all the jar files packed in the web archive and dumped an output of the fully qualified class along with the jar file that it can be found in and a count of the number of occurrences. Here's the ruby script running on a Windows platform:
</p>
<pre class="brush: ruby;">
java_home="C:\\sdk\\jdk1.6.0_24\\bin"
classmaps = Hash.new(0)
Dir["*.jar"].each do |file|
cmd = sprintf("%s\\jar -tf %s",java_home,file)
lines = `#{cmd}`
lines.each do |line|
if line =~ /.class/
key = line.chomp
if classmaps.key?(key) then
old = classmaps[key]
data = sprintf("%s,%s",old,file)
classmaps[key] = data
else
classmaps[key] = file
end
end
end
end
classmaps.each do |k,v|
tokens = v.split(",")
printf("%s,%i,%s\n",k,tokens.size,v)
end
</pre><p>I then loaded the output file in csv format into Excel and sorted the occurrences in descending order and was flabbergasted to find numerous entries that look something like the following entry:</p>
<table>
<tr><th>Class Name</th><th>Occurrence</th><th colspan=6>Found in Jar files:</th></tr>
<tr>
<td>kodo/jdo/KodoExtent.class</td><td>6</td><td>mylib.jar</td><td>mylib.jar</td><td>mylib.jar</td><td>kodo-api.jar</td><td>kodo-runtime.jar</td><td>kodo.jar</td>
</tr>
</table>
<p>Somehow, for this application team, they were able to add the same class to the same jar file multiple times. I never thought that it was possible to add the same class to the same jar file multiple times nor would I ever want to do that. When I finally confronted the application team about this, they recognize that their build process is broken and need to fix their build process. But as a quick test, the developer removed some of the duplicate classes related to the library upgrade and the problems went away. Until Java 8 is introduced with Modularity capabilities, this tool has will be a handy way for me to check duplicate classes given a list of jar files.</p>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com1tag:blogger.com,1999:blog-18281936.post-74055200431200600732012-01-20T17:52:00.000-08:002012-01-20T17:52:37.015-08:00Translating If-Then-Else Control Flow Idiom to F#<p>I was reading through Juval Löwy's <a href="http://www.amazon.com/exec/obidos/ASIN/0596805489/techie2biz-20">Programming WCF Services</a> book and wondering if I should do a series of WCF blog posts in F# based on Löwy's book when I ran into a common construct found in C# programs. That construct looks something like the following C# code:
</p>
<pre class="brush: csharp;">
public static void MyMethod(String oldstuff, String newstuff, bool flag)
{
if (oldstuff == null)
throw new Exception("oldstuff is null!");
if (newstuff == null) {
DoSomething("Default");
return;
}
if (flag == false) {
DoSomething(oldstuff);
return;
}
DoSomething(newstuff);
}
</pre>
<p>This is a construct that I oftened have used in the past and have never thought about it much. But when you translate the above code directly into F#, it becomes a lot more verbose because F# requires you to implement the <code>then</code> clause. A direct translation to F# as follows:</p>
<pre class="brush: fsharp;">
let mymethod oldstuff newstuff flag =
if oldstuff = null then
raise (new Exception("oldstuff is null!"))
else
if newstuff = null then
DoSomething("Default")
else
if flag = false then
DoSomething(oldstuff)
else
DoSomething(newstuff)
</pre>
<p>If I had a lot of these if-then-else statements in my C# method, then my F# version would disappear off the screen to the right if I tried to implement it by direct translation. I thought about how I could implement this in F# and came up with this following possibility:</p>
<pre class="brush: fsharp;">
let revised_mymethod oldstuff newstuff flag =
let (_,action) =
[(oldstuff=null, lazy (raise (new Exception("oldstuff is null!"))));
(newstuff=null, lazy (DoSomething("Default")));
(flag=false, lazy (DoSomething(oldstuff)));
(flag=true, lazy (DoSomething(newstuff)))]
|> List.filter fst
|> List.head
action.Force()
</pre>
<p>Rewriting the C# code in this fashion makes me think of rules engines and after refactoring out some common code, I could rewrite the above F# code as follows:</p>
<pre class="brush: fsharp;">
let followrules (xs:(bool*Lazy<unit>) list) =
(xs |> List.filter fst |> List.head |> snd).Force()
let revised_mymethod2 oldstuff newstuff flag =
[(oldstuff=null , lazy (raise (new Exception("oldstuff is null!"))));
(newstuff=null, lazy (DoSomething("Default")));
(flag=false, lazy (DoSomething(oldstuff)));
(flag=true, lazy (DoSomething(newstuff)))]
|> followrules
</pre>
<p>With this new construct, I can easily re-arrange the order of evaluation, add or remove new conditions. This new construct just seems to have more advantages than the old if-then-else construct in F#.</p>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com4tag:blogger.com,1999:blog-18281936.post-70013457176671088702012-01-11T10:32:00.000-08:002012-04-05T14:45:10.758-07:00Testing Coherence with Clojure<script type="text/javascript" src="https://raw.github.com/sattvik/sh-clojure/master/src/shBrushClojure.js"></script>
<p>A developer came to me the other day asking for help in diagnosing some issues with their application and the interaction with Oracle's Coherence product. I wanted to write some testing harness to quickly test some Coherence configuration and gave some thought about how I would go and try to replicate the issues that the application had. I wanted a REPL environment so that I can interactive manipulate the Coherence API and dump outputs on demand. I decided to use Clojure to experiment with Coherence, although I could have used JRuby, Jython, Groovy or Scala. From purely a familiarity perspective, I would rank my usage of these listed languages in the order of Ruby first, Python second, Groovy third, Clojure fourth and Scala last. But for some unknown, deep-seated and probably emotional reasons, I like Clojure more and relish the opportunity to use it. One of the first thing I tried with Clojure and Coherence is to perform a timing test on adding data to a 2 node distributed cache in serial vs concurrent mode. Here's the example Clojure script:</p>
<pre class="brush: clojure;">
(import '(org.apache.commons.lang3 RandomStringUtils)
'(java.math BigInteger)
'(java.util Random Date HashMap)
'(com.tangosol.net NamedCache CacheFactory CacheService Cluster))
(CacheFactory/ensureCluster)
(def cache (CacheFactory/getCache "sandbox"))
(defn random-text [] (RandomStringUtils/randomAlphanumeric 1048576))
(defn random-key [] (RandomStringUtils/randomAlphanumeric 12))
; Testing serial puts
(new Date)
(dotimes [_ 200] (.put cache (random-key) (random-text)))
(new Date)
; Testing concurrent puts
(def buffer (new HashMap))
(new Date)
(dotimes [_ 200] (.put buffer (random-key) (random-text)))
(.putAll cache buffer)
(new Date)
</pre>
<p>Running this code showed a 2x gain in speed of data load. On one of the Coherence nodes, I had JVisualVM connected to it and watched the realtime GC behaviors with VisualGC. It has been fascinating to watch the behaviorial differences between serial vs parallel data load and the memory activities of the Coherence node when my Clojure script was idle. I hope to conduct more tests in the future looking at GC behaviors and leverage my Clojure script as load driver in my testing efforts and assist me in GC tuning of Coherence instances.</p>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com0tag:blogger.com,1999:blog-18281936.post-42918559590707549302011-11-11T16:41:00.000-08:002011-11-11T16:41:58.905-08:00Revisiting the SharePoint collection adapter for F#<p>Recently, I got to work with SharePoint 2007 again and I revisited a previously written blog <a href="http://jyliao.blogspot.com/2008/09/exploring-sharepoint-2007-object-model.html">Exploring SharePoint 2007 Object Model with F#</a>. In that blog, I complained about the fact that SharePoint collections did not implement IEnumerable, causing me to implement a type specific collection adapter as pilfered from <a href="http://asadewa.wordpress.com/2008/01/03/linq-ing-splistcollection/">Asfar Sadewa</a>. However, as I started to work with more different SharePoint collections, I realized that I need a generic SharePoint collection adapter. But once again, I was foiled by SharePoint's library designers. I wanted to ensure some level of type safety for my generic collection adapter and wanted to bound the collection item generic type variable so the client code is restricted to types such as <code>SPWeb</code>, <code>SPList</code>, etc. Looking up the hierarchy, I see the type <code>SPSecurableObject</code>. Unfortunately, SharePoint library designers did not make <code>SPSecurableObject</code> visible to others and the next level up that hierarchy is the Object type. So I'm stuck with an unbounded generic type for the collection item in my generic version SharePoint collection adapter. Actually, what I really want is to ensure that the collection item type is consistent with the collection type. I haven't quite figure out if that's even possible to apply that kind of type parameter constraints in C#. Here is generic version of the SharePoint collection adapter (with the runtime type error leakages) implementation:
</p>
<pre class="brush: csharp;">
using System;
using System.Collections.Generic;
using Microsoft.SharePoint;
namespace SharePoint.Utility
{
// I really wanted to apply constraint to TPart to SPSecurableObject
public class SPCollectionAdapter<TPart, TCollection> : List<TPart> where TCollection : SPBaseCollection
{
private TCollection _listCol;
public SPCollectionAdapter(TCollection listCol)
{
_listCol = listCol;
Refresh();
}
private void Refresh()
{
this.Clear();
this.Capacity = _listCol.Count;
foreach (TPart item in _listCol)
{
this.Add(item);
}
}
}
}
</pre>
<p>This allows me to write my F# code as follows:</p>
<pre class="brush: fsharp;">
open Microsoft.SharePoint
open SharePoint.Utility
let path="http://localhost/"
let collection = new SPSite(path)
let site = collection.RootWeb
let lists = SPCollectionAdapter<SPList,SPListCollection>(site.Lists)
Seq.iter (fun (x:SPList) -> printf "%s\n" x.Title) lists
</pre>John Liaohttp://www.blogger.com/profile/04740715435312568366noreply@blogger.com2