Tracking test results from TestNG

The development team wants fast feedback on the latest release and you have a large suite of (TestNG) integration tests to run, analyze and report back on. Running the tests is usually the easy bit but without an automated approach to diff results against previous results you may miss a new or regression bug. To reduce the developer feedback response time I searched for a (open source) tool that would allow:

1. Centralized tool to persist, view and perform analysis on testng result history
2. Integrate smoothly with TestNG i.e. implemented http://testng.org/javadocs/org/testng/ITestListener.html
3. Test runs from gradle command line
4. Fast identification of changes in test results from previous test run, release etc
5. Ability to track a single test over a number of builds

I found such a tool called cuanto.  The tool hasn’t been updated in a while but it works and the source is available on github. The setup was straightforward Win2008 server, MySQL and Jetty.  One minor hiccup is you cannot use the latest version of java with Jetty; I used 1.6_45. Setup:

1. Install MySQL and create a database called ‘cuanto’
2. Install jetty (uncompress it)
3. Uncompress the cuanto-2.8.0.war into /jetty/webapps/cuanto
4. Edit cuanto-db.groovy as per the INSTALL doc and drop it into  /jetty/webapps/cuanto/WEB-INF/classes
5. Download mysql java driver (mysql-connector-java-5.1.28-bin) and drop into /jetty/webapps/cuanto/WEB-INF/lib
6. Start server /jetty/java -jar start.jar
   1. The first time the server is launched it will create the tables in cuanto database
7. Test with http://<servername&gt;:8080/cuanto

So, the tool is setup and we move onto integrating it with Gradle build.gradle.  Cuanto comes with a rest api that the listener jar uses to pass results to the jetty server. Reminds me of a similar approach I took to persist HP QTP (now called Unified testing tool) test results to a Sybase database back in BlackRock days.  Happy to have moved on from both ;-). The Gradle code snippet is below:

task cuantotest (type: Test){
 systemProperty "cuanto.url", "http://SERVERIPHERE:8080/cuanto"
 systemProperty "cuanto.projectkey", "key"
 systemProperty "cuanto.testrun.create", "true"

 useTestNG()<{
   useDefaultListeners = false
   ignoreFailures = true
   includeGroups 'test1'
   <strong>listeners << 'cuanto.adapter.listener.testng.TestNgListener'</strong>
  }
 }

 repositories {
   flatDir {
     dirs 'e:/tools/cuanto-2.8.0/adapter'
   }
 }

Thats it. run your task: gradle cuantotest and view your results on cuanto site.  I used the create test run id automatically option.

As this was a test I called the cuanto jar files locally hence the gradle files repo call; unfortunately the latest jars are not in maven central. You can always add those to your own organizations repository. A slightly better way to implement this is to use gradle’s files dependencies:

dependencies {
  runtime files('e:/tools/cuanto-2.8.0/adapter/cuanto-api-2.8.0.jar', 'e:/tools/cuanto-2.8.0/adapter/cuanto-adapter-2.8.0.jar')
  //added for cuanto
  runtime 'commons-httpclient:commons-httpclient:3.1'
  runtime 'net.sf.json-lib:json-lib:2.4:jdk15'
  //testng
  compile 'org.testng:testng:6.8.7'
  ...
  ...
}

Last thought is think about how you want to structure your test results within cuanto from a group, project and test run perspective.

Stuff you need (all latest versions apart from Java):

1. TestNG
2. Gradle
3. Cuanto  (trackyourtests.com)
4. Webserver (Jetty)
5. Database (MySQL)
6. Java