Category Archives: testing

TestNG – How to create a single report for tests in gradle

If you are using testng in a gradle multi (sub) project scenario and want to centrally locate all your TestNG html reports into one folder, this is how I did it with latest versions of TestNG and Gradle (1.8)

/////////////////////////////////////////////////////////////////////////////////////
apply plugin: ‘java’

subprojects{

//override the java plugin test task
test {
//runs tests across all sub-projects. good for CI build process.
ignoreFailures = true
//TestNG support in gradle java plugin
useTestNG()
}

}

//new task, outside of the subprojects task, which creates a top level directory for all of your tests in sub projects
task testReport(type: TestReport) {
//
destinationDir = file(“reports/test”)
subprojects {
reportOn { tasks.withType(Test) }
}

}
/////////////////////////////////////////////////////////////////////////////////////

Advertisements

The joy of JMeter Part 1

JMeter is an opensource testing tool that I have used recently and I felt like putting together some thoughts on its use and   some general performance testing guidelines on what I’ve seen to be successful. I’m going to avoid the opensource versus vendor tool discussion having successfully used HP, Borland, Parasoft tools and most recently JMeter.  Every organization and situation is different but the tool choice needs to take into account factors such as sdlc, release management process, technology stack, product (b2b/b2c) or internal app, team dynamics etc. For example, consider testing a mobile app as opposed to an internal trading system.

My experience lends itself to testing prior to production release but a growing trend in performance testing involves running performance testing in the production environment, leveraging a set of ‘cloud’ based assets. One company I like and have talked to is SOASTA, they a great group of people plus have developed a very impressive testing solution both from a GUI design and also feature set perspective.  This is a powerful and useful technique and hopefully I can get some exposure to this soon…

First off you need to plan, plan, plan. Really this is the most important aspect of any testing effort (and anything really worthwhile doing in life too…).   If you hear “well lets test until it breaks, that’s easy isn’t it?” I advise running for the hills! If that isn’t possible then getting everyone together and deciding on what the objectives are will really pay off when you are in the ‘throws of testing’ the release.

Here are what I see as some high-level discussion points for upcoming posts on this thread:

  • Performance Objectives
  • Test scenario and data design
  • Monitoring
  • Functional test while you go!
  • Know thy (technology) stack!
  • JMeter + plugins
  • JMeter sampler development
  • Test environment
  • Outsourcing your test lab
  • Running the tests
  • Compiling the run data
  • Reporting
  • Injecting performance testing to your sldc

Part 2 to follow on setting the performance objectives, which will drive the entire effort.

quality signals…

an interesting post on testing 2.0 from the folks over in google; using signals from various sources to predict code quality. I’ve been thinking about how to create a picture of composite quality from sources like sonar (static analysis) and testing (defect density, defects before and after release)

anyway, link to google testing blog below

http://googletesting.blogspot.com/2012/08/testing-20.html

manual and automated testing – bringing it all together

I’ve heard the following on occasion “can’t you just start automating?” Well I could but I won’t.

What I want to do in this blog is to describe the symbiotic nature of both practices; yes manual testing and automation although different skillsets are very much related and both need each other to be successful.

Lets look at what manual testing is. First off, the term itself is terrible as it implies a rote, tedious and laborious activity (If you have a better term let me know!).  This is so far from reality as the practice requires:

  1. Subject matter expert level of knowledge about the business domain, application(s), data, architecture and of course the systems being tested
  2. Analytical skills to take a disparate set of documentation (use cases,tech specifications etc) and turn it into detailed and repeatable tests
  3. Discipline to document tests and keep them maintained release after release
  4. Methodical approach to running tests manually (oh that’s where its comes from!) and capturing results with the objective of creating useful and actionable bug reports

Why does manual testing need automation? executing thousands of tests manually would take far too long and be too error prone.  The trick to it is to have the correct balance between manual and automated test execution; leveraging the best of both worlds to deliver test turnaround times in a fast and efficient manner.

ok lets look at automation. Go ahead call an automation engineer a ‘scripter’ see the eyes roll. Creating automation is a development exercise and to be successful requires expertise and knowledge on par with a developer-developer (if you know what I mean).  In fact, I think thats why automated testing has failed many times and has somewhat of a bad rep that it does not get treated with the same care and attention as software development. Allied with vendors pushing tools “record & playback so you will never need to code” syndrome.

Why does automation need manual testing?  The manual tests provide the specification for the automation developer to follow when performing the initial implementation and subsequent maintenance of the automated test. Starting to automate without (well exercised) manual tests in place is not a good practice and leads to eventual abandonment. I look at automated tests as the tip of the iceberg and all of the preceding steps (requirements analysis, q&a sessions, hard won biz knowledge, test case creation and maintenance etc) that the tester performs.

So please keep this in mind when a tester is asking for clarifications or more information!