Tracking test results from TestNG

The development team wants fast feedback on the latest release and you have a large suite of (TestNG) integration tests to run, analyze and report back on. Running the tests is usually the easy bit but without an automated approach to diff results against previous results you may miss a new or regression bug. To reduce the developer feedback response time I searched for a (open source) tool that would allow:

  1. Centralized tool to persist, view and perform analysis on testng result history
  2. Integrate smoothly with TestNG i.e. implemented
  3. Test runs from gradle command line
  4. Fast identification of changes in test results from previous test run, release etc
  5. Ability to track a single test over a number of builds

I found such a tool called cuanto.  The tool hasn’t been updated in a while but it works and the source is available on github. The setup was straightforward Win2008 server, MySQL and Jetty.  One minor hiccup is you cannot use the latest version of java with Jetty; I used 1.6_45. Setup:

  1. Install MySQL and create a database called ‘cuanto’
  2. Install jetty (uncompress it)
  3. Uncompress the cuanto-2.8.0.war into /jetty/webapps/cuanto
  4. Edit cuanto-db.groovy as per the INSTALL doc and drop it into  /jetty/webapps/cuanto/WEB-INF/classes
  5. Download mysql java driver (mysql-connector-java-5.1.28-bin) and drop into /jetty/webapps/cuanto/WEB-INF/lib
  6. Start server /jetty/java -jar start.jar
    1. The first time the server is launched it will create the tables in cuanto database
  7. Test with http://<servername&gt;:8080/cuanto

So, the tool is setup and we move onto integrating it with Gradle build.gradle.  Cuanto comes with a rest api that the listener jar uses to pass results to the jetty server. Reminds me of a similar approach I took to persist HP QTP (now called Unified testing tool) test results to a Sybase database back in BlackRock days.  Happy to have moved on from both ;-). The Gradle code snippet is below:

task cuantotest (type: Test){
 systemProperty "cuanto.url", "http://SERVERIPHERE:8080/cuanto"
 systemProperty "cuanto.projectkey", "key"
 systemProperty "cuanto.testrun.create", "true"

   useDefaultListeners = false
   ignoreFailures = true
   includeGroups 'test1'
   <strong>listeners << 'cuanto.adapter.listener.testng.TestNgListener'</strong>

 repositories {
   flatDir {
     dirs 'e:/tools/cuanto-2.8.0/adapter'

Thats it. run your task: gradle cuantotest and view your results on cuanto site.  I used the create test run id automatically option.

As this was a test I called the cuanto jar files locally hence the gradle files repo call; unfortunately the latest jars are not in maven central. You can always add those to your own organizations repository. A slightly better way to implement this is to use gradle’s files dependencies:

dependencies {
  runtime files('e:/tools/cuanto-2.8.0/adapter/cuanto-api-2.8.0.jar', 'e:/tools/cuanto-2.8.0/adapter/cuanto-adapter-2.8.0.jar')
  //added for cuanto
  runtime 'commons-httpclient:commons-httpclient:3.1'
  runtime 'net.sf.json-lib:json-lib:2.4:jdk15'
  compile 'org.testng:testng:6.8.7'

Last thought is think about how you want to structure your test results within cuanto from a group, project and test run perspective.

Stuff you need (all latest versions apart from Java):

  1. TestNG
  2. Gradle
  3. Cuanto  (
  4. Webserver (Jetty)
  5. Database (MySQL)
  6. Java

Getting TestNG results reporting in Bamboo (in a Gradle build)

To get TestNG parser working in Bamboo do the following:

  1. In top level build.gradle
      • use ‘useDefaultListeners=true’ this produces the testng-results.xml file(s)*
      • Put everything in a code block under useTestNG()
  1. use ant style **/testng-results.xml as the file locator  in Bamboo’s

here is the build.gradle:
test {
useDefaultListeners = true
ignoreFailures = true

* This works for a multi-project gradle setup as the testng.xml files are located in each sub-directory.

Answered my own problem on Atlassian site

TestNG – How to create a single report for tests in gradle

If you are using testng in a gradle multi (sub) project scenario and want to centrally locate all your TestNG html reports into one folder, this is how I did it with latest versions of TestNG and Gradle (1.8)

apply plugin: ‘java’


//override the java plugin test task
test {
//runs tests across all sub-projects. good for CI build process.
ignoreFailures = true
//TestNG support in gradle java plugin


//new task, outside of the subprojects task, which creates a top level directory for all of your tests in sub projects
task testReport(type: TestReport) {
destinationDir = file(“reports/test”)
subprojects {
reportOn { tasks.withType(Test) }


ALM Jenkins Plugin

HP have released a Jenkins plugin for ALM, UFT, QTP and ST versions 11 and up.

This is a good move on HPs part – allowing automation to be run as part of a CI (or even a CD) process. I do think the outcome would be better for web/service testing as opposed to GUI testing, which does not typically support non-breaking changes like e.g. webservices do.

This does go some ways to increasing the value proposition for HP in the agile development process. But I do have to wonder if you are testing rest or webservices in conjunction with a web GUI would you even be looking at ALM, UFT or QTP as a solution?