A short while back I blogged on my thoughts-from-the-front-line-with-thoughtworks-twist and mentioned in it that it provided a pretty good reporting tool in its own way. Well we managed to put it to good use. Ok, we hacked it a bit, but I will tell you what we did, and if you use this package (v4.2 I think it is), you can do the same thing, and the result is pretty impressive. (For later versions I cannot imagine they’ve refactored everything so you could probably do the same stuff).
Twist is really designed as a BDD tool for performing BDD tests, and it uses Selenium (amongst others) for testing its UIs. However I had my own problems.
As an Environments Manager
I want to know how all my environments are performing
So that I can respond before a problem is noticed by the users
or perhaps in BDD form
Given we have problems in environments
When an environment has an issue
Then I want to be 2 steps ahead of the users and fix it already
The problem was quite simple, and common, we had no-one checking the environments regularly, and any issues that the users faced were not being resolved very quickly. After some analysis we frequently found that the problem had been there for some time; perhaps the server was down at midnight but the users noticed at 11am. Had we known about the issue we could have had it fixed at 9.15, but the issue was only exposed when the users tried to test the system. (ps. User == customer === Business Analyst). Manually checking these environments out of hours is really an inefficient (and tedious) use of people’s time, and that is completely out of the question.
So All we needed to do is point Twist at the environments. Using Selenium to impersonate a user, log in, and perform simple functions we were able to identify a huge percentage of problems before the users noticed. We would schedule Twist so it would run every 15 minutes thus providing a heartbeat of all of the environments.
However the key hurdle is this, how do we display the results, and to whom. Agile is a great advocate of transparency, and we so happened to have a ceiling mounted 40″ monitor where everyone could see what was happening. All we needed is display the results on the monitor. Which meant we needed the results in a nice displayable easy-to-feedback format, a proper information radiator, with an appropriate meme. That everyone could see.
As you know Twist has its own rich functioning report page. But it was too “busy” for our purpose. Somewhere in the field of jar files was a bit of CSS and HTML generating the report… and we found it.
In the midst of the plugins directory was this jar file: com.thoughtworks.twist.core_2.4.0.12641-3bffc171c71291.jar
In the midst of the jar file was a handy file called report.vm (com\thoughtworks\twist\core\execution\report). This file is essentially an Apache Velocity templated file. This produced the html and we edited it to produce a simple table showing the successful execution of the end to end tests. Because we could, we managed to add in a few customised Internet memes, randomly selected based upon the severity of the state of the environments.
For a while people came along to stare at the screen, because the memes were amusing. But the average time of an environment becoming unresponsive/crashing/experiencing an outage and the Environments Team noticing changed from (sometimes) hours to minutes. We were also able to identify outage patterns with this and remove external dependencies that caused them.
Close out your feedback loops
The biggest lesson here, is my favourite lesson of feedback loops. Close out your feedback loops. What you don’t want to have in any project is a feedback loop which is long and arduous. There will always be problems, lets not focus on planning to avoid them, but dealing with them in a swift and dexterous way.
One argument I did think of against using Twist for this purpose was that I could have just used “Selenium” to do this and generated my own HTML. However, this fails for two reasons, the main one being, the kinds of tests BDD/Plain Language style we can insert at short notice allows us to really enrich the type of testing that we do. So this is not just to see if a page is responding or not, but also to test basic functionality and look for targeted errors. This is really easy to execute and maintain in twist using their provided scenarios and eclipse style implementation of the tests, separating business value (or testing value) from the executed tests.
If you can picture it, a new visual bug is found by the test team in the morning, and can be included in the tests and checks for all the environments in the afternoon, or perhaps the bug fixes found last month, are they included in this particular environment? its that easy to do.
One Response to Thoughtworks Twist as a reporting tool – feedback loops part 1