Testing Coherence with Clojure

A developer came to me the other day asking for help in diagnosing some issues with their application and the interaction with Oracle's Coherence product. I wanted to write some testing harness to quickly test some Coherence configuration and gave some thought about how I would go and try to replicate the issues that the application had. I wanted a REPL environment so that I can interactive manipulate the Coherence API and dump outputs on demand. I decided to use Clojure to experiment with Coherence, although I could have used JRuby, Jython, Groovy or Scala. From purely a familiarity perspective, I would rank my usage of these listed languages in the order of Ruby first, Python second, Groovy third, Clojure fourth and Scala last. But for some unknown, deep-seated and probably emotional reasons, I like Clojure more and relish the opportunity to use it. One of the first thing I tried with Clojure and Coherence is to perform a timing test on adding data to a 2 node distributed cache in serial vs concurrent mode. Here's the example Clojure script:

(import '(org.apache.commons.lang3 RandomStringUtils) 
        '(java.math BigInteger)
        '(java.util Random Date HashMap)
        '(com.tangosol.net NamedCache CacheFactory CacheService Cluster))

(CacheFactory/ensureCluster)
(def cache (CacheFactory/getCache "sandbox"))

(defn random-text [] (RandomStringUtils/randomAlphanumeric 1048576))
(defn random-key [] (RandomStringUtils/randomAlphanumeric 12))


; Testing serial puts
(new Date)
(dotimes [_ 200] (.put cache (random-key) (random-text)))
(new Date)

; Testing concurrent puts
(def buffer (new HashMap))
(new Date)
(dotimes [_ 200] (.put buffer (random-key) (random-text)))
(.putAll cache buffer)
(new Date)

Running this code showed a 2x gain in speed of data load. On one of the Coherence nodes, I had JVisualVM connected to it and watched the realtime GC behaviors with VisualGC. It has been fascinating to watch the behaviorial differences between serial vs parallel data load and the memory activities of the Coherence node when my Clojure script was idle. I hope to conduct more tests in the future looking at GC behaviors and leverage my Clojure script as load driver in my testing efforts and assist me in GC tuning of Coherence instances.

Revisiting the SharePoint collection adapter for F#

Recently, I got to work with SharePoint 2007 again and I revisited a previously written blog Exploring SharePoint 2007 Object Model with F#. In that blog, I complained about the fact that SharePoint collections did not implement IEnumerable, causing me to implement a type specific collection adapter as pilfered from Asfar Sadewa. However, as I started to work with more different SharePoint collections, I realized that I need a generic SharePoint collection adapter. But once again, I was foiled by SharePoint's library designers. I wanted to ensure some level of type safety for my generic collection adapter and wanted to bound the collection item generic type variable so the client code is restricted to types such as SPWeb, SPList, etc. Looking up the hierarchy, I see the type SPSecurableObject. Unfortunately, SharePoint library designers did not make SPSecurableObject visible to others and the next level up that hierarchy is the Object type. So I'm stuck with an unbounded generic type for the collection item in my generic version SharePoint collection adapter. Actually, what I really want is to ensure that the collection item type is consistent with the collection type. I haven't quite figure out if that's even possible to apply that kind of type parameter constraints in C#. Here is generic version of the SharePoint collection adapter (with the runtime type error leakages) implementation:

using System;
using System.Collections.Generic;
using Microsoft.SharePoint;

namespace SharePoint.Utility
{
    // I really wanted to apply constraint to TPart to SPSecurableObject
    public class SPCollectionAdapter<TPart, TCollection> : List<TPart> where TCollection : SPBaseCollection
    {
        private TCollection _listCol;

        public SPCollectionAdapter(TCollection listCol)
        {
            _listCol = listCol;
            Refresh();
        }

        private void Refresh()
        {
            this.Clear();
            this.Capacity = _listCol.Count;

            foreach (TPart item in _listCol)
            {
                this.Add(item);
            }
        }

    }
}

This allows me to write my F# code as follows:

open Microsoft.SharePoint
open SharePoint.Utility

let path="http://localhost/"
let collection = new SPSite(path)
let site = collection.RootWeb
let lists = SPCollectionAdapter<SPList,SPListCollection>(site.Lists)
Seq.iter (fun (x:SPList) -> printf "%s\n" x.Title) lists

Porting the Log Analysis Code to Haskell

My coworker approached me the other day and ask what open source log analysis tools would I recommend. I personally do not have much experience with a general purpose open source log analysis tools so I would have probably recommended him to take a look at Splunk. Since I've recently written a customized log analysis software, I became curious and asked him what he intend to do with the log analysis software.

My coworker said that he needed to analyze Tibco EMS logs. Tibco EMS logs incoming messages in the order it receives. My coworker is interested in a set of related messages that is identified by the message ID tag. His particular issue is that the logs entries that he's interested in are interspersed with other log entries that he's not interested. He wanted a log file where the log entries are grouped by message id in historical order.

Once I understood his needs, I realize that he did not need the Splunk and that I could quickly adapt my F# log analysis software written in the previous blog post for his need. When I gave him the modified F# code, he asked me if I could port it to Linux. That threw me for a loop. I briefly entertained the idea of building a Mono system and compile F# on Mono but decided against it for now. I thought it would be easier to just port it to Haskell, which I already have on Linux.

Here's the ported Haskell log analysis software with modifications to work with Tibco log entries.

import Data.Time.Calendar
import Data.Time.LocalTime
import Data.Time.Parse
import List
import System.Environment

type Category = String
type Entry = [String]
type TimeStamp = (LocalTime,String)
type LogHeader = (TimeStamp, Category)

alphaTime = LocalTime (fromGregorian 2000 1 1) midnight 

data LogEntry = LogEntry (TimeStamp, String) [String]
                deriving (Show)

{- Grab label -}
categorize (_ : _ : label : _) = label
categorize words = ""

{- Grab timestamp -}
timestamp (date : time : _ )  = strptime "%Y-%m-%d %H:%M:%S" (date ++ " " ++ time)
timestamp words  = Nothing
           
{- header :: String -> (String, Maybe (LocalTime, String)) -}           
header line = (timestamp tokens, categorize tokens)    
    where tokens = words line

{- Concrete implementation of Tibco log parser -}    
logparser :: [String] -> LogHeader -> [String] -> [LogEntry] -> [LogEntry]
logparser (line : rest) xheader entry entries  = process (header line)    
    where process (Just (ts),label) = 
              logparser rest h [line] ((LogEntry xheader (reverse entry)):entries ) where h = (ts,label)
          process (Nothing,_) = logparser rest xheader (line : entry) entries

logparser [] xheader entry entries = reverse ((LogEntry xheader entry) : entries) 

{- Utility method  to pull items out of LogEntry -}
entry (LogEntry _ entries) = entries    
category (LogEntry (_,label) _) = label

{- comparator based on category -}
categorysort (LogEntry (_,a) _) (LogEntry (_,b) _) 
    | a > b = GT
    | a < b = LT
    | otherwise = EQ

parselog parser lines = parser lines ((alphaTime,".000"),"STARTFLAG") [] []
    
processlog = unlines
             . map (unlines . entry)
             . sortBy categorysort
             . (parselog logparser)
             . lines 

main = do (filename:_) <- getArgs
          contents <- readFile filename
          putStr (processlog contents)