Model based testing - the future?

So there is this thing: model based testing... sometimes they call it property based testing. The concept is simple, instead of writing test cases (which can be really, really inefficient) you just tell to a droid what to do. Of course it is not that simple... the droids nowadays requires a lot of explanations...

In nutshell it means the following: you create a finite state machine which is the expected behavior of you application, and based on that, you can generate tests.
The edge describe the action what you need to do with the system under test, and the node is the expected state, so the content of your assertion. Sounds simple...

My previous experiences with model based testing were...

• If there is any tool support for visualizing and executing, storing, creating, whatever, it is really expensive, and it tells you the way how you can interact with the system....  you are strongly binded to the tool.
• After a while, when you try to cover all your requirements with the model you end up with creating a huge, unmaintainable monster, what is more difficult to understand than the implemented system, and the hair stand up on the back of your neck if you just think that you have to change it
• so you decide to throw it away and implement tests based on your requirements and your inspiration, with a few comments, so everyone can understand what is the feature/method about...

But what do we have now?

In the last few months I started to use two tools... well I am not extremely experienced in any of them, but at least I am familiar with the features it can give to a QA engineer. Both tools are similar in a few points:

• you use a programming language to implement your model based test
• so you have to do your own FSM implementation
• the tool provides you the bricks what you can use to build up your model:
• a transaction/action where you call a method to change the state of the system under test (aka test call)
• a post condition where you can check is the state of your application is what you expect (after the log in I have a proper session)
• a precondition/guard where you can filter those transactions out what you do not want to call with the given state (logout when you are not logged in)
• next state: where you can change your state (it only exists in one tool, in the other you have to put it in the transaction or in the post condition)
• and the tool gives you other features to make the execution better (on this level, they are really different)

So the bricks are simple. A lot depends on you, and how you implement this thing. And here the KISS principle is critical.

And there is another thing about model based testing. As a tester we want to think on end-to-end level. We use the requirements to create tests, we try to put ourselves in the user's/customer's mind, we want to discover those cases when something can go wrong. And it is hard to align these things with model based testing...

Why?

Sometimes it is just too much for a model, you must keep it simple!

So I can imagine two strategies to create the model:

1. Find that area in you application which is the most critical, but relatively small. And apply an extensive model for that (so you have to reduce your scope, to not have a large model).
1. Benefit? your model won't be complex, and your critical part will be covered. And do not forget, you can have multiple models for different aspects/parts...
2. Step back, and view the whole system, and collect those interactions what your users will normally do with it, and create a model based on that. So basically you connect the happy paths together, to see if with a long, diverse usage, your application is still flawless. 
1. Benefit? your model is not complex, because your functional tests will focus on the edge cases, and other parts of the requirements, what is to expensive to cover here.

So that's it for today, in the next posts I will introduce the tools for you, and what I tried (and failed), and finally done (and doing) with it.

Sonar and Erlang... what have I done?!

It started long time ago...
After a week on Dagobah (endless rain) where I spent my holidays with my little clones and my wife, it is time to tell you my last great(?) achievement. BTW next year we go to a more sunny place like Tatooine (maybe too sunny) or Naboo (somewhere in the middle). But go back to the topic:

everything was started with my transfer to a new battleship, I have to start this from the beginning otherwise I will lost the thread. So as a home project, while my family went back to Naboo or a place like that, I was able to work on it at home as well. The problem with adding something to Sonar, the lack of documentation. There are no step-by-step howtos, how to use the violation part, or how to create measures and so on, but there are a plugins what you can use as a reference... The problem is with that, that every plugin uses the same (or older, deprecated) methods in a different way... it means that the copy-paste not always a good idea, but at least a good starting point. My dear old ex-fellow Alex was a real copy-paste programmer, I hope he is still fighting for the Republic! SO I have no problem with copy-pasting in this case, otherwise you cannot make it work, and you cannot try out code snippets.

Today the plugin is really cool. Next to the basic static analysis (lines of code, number of lines, comments, comment density, number of files, methods, exported methods, public documented APIs, etc...), I could import eunit results (unit test success, unit test coverage), and metrics from an erlang analyzer tool, (what was created by people from the same cloning machine where I was created) like complexity, cohesion... (some of these: http://pnyf.inf.elte.hu/trac/refactorerl/wiki/MetricQuery) so now you can get a quite good overall picture about your erlang software. It also has violation reports, based upon dialyzer and refactorerl things.
So I have reached the state where it is not too easy to extend the plugin...the question, what else do we want? Is it anything more what we can do? Well sometimes I am a compulsive guy, and I cannot let go what I have doubts about, what I don't feel complete. There are some obvious part: IT test, and its coverage... the problem here, is what the hack is IT test? To me, in my term the IT test can be something like acceptance test, so not just IT testing in small, but in big, or you know what I mean. Now it is not clear how to measure coverage for the second phrase, so I think I will implement the IT testing in small cover results... this is one... the other one, or two is about refactorerl... it has some metric what are not close to violations, but real metrics, what we (maybe) want to measure and display (number of imported modules, number of OTP modules and so on...) These things needs a widget to display them... I have started to work on it, but it is a completely different part of sonar, so I have to start over...
Another thing is that refactorerl can generate exciting graph about your code, about dataflow, method calls, and a few more. These are really cool, but to big for a simple code, so I have to find a way to generate something which is smaller and we can display it in another widget...

So this is what we have, now it needs some advertisement, maybe others can use it as well...

oh I forget about the limitations...
the first one it is a little bit against the sonar concept, but it has really strong dependencies for external applications, like dialyzer and refactorerl... but this is the way how it works, there is no chance to remove them with reasonable effort.
It is strongly works in the way how we build and think about our erlang applications... maybe it cannot work on other stuffs of other corps...

so here is a picture about an erlang app (with submodules) dashboard:

if you want it, come and get it

my next post will be about some technical things how we actually analyse and call sonar on our projects (three different way)

Quality measurement part 1

My Colonel asked my Regimental Commander who asked my Master Chief to find out how to measure the Quality of a Software. I know it is not unique, and there are hundred of pages and calculations out there in the Universe, but my mind collapsed when it tried to find out how.

I went through all phases of protesting: laughing, crying, bargaining...
We had to create a machine which eats numbers and splits a number, a beautiful, magical number which is comparable, descriptive, short and tells everything about quality.

In one of my stations I tried to make fun of it, to show its incongruity. I created metrics like:

• # of [public methods | lines of code | methods ] / years of experience of the team

It is definitely about quality, shows the years of experience index of each method... (of course with the presupposition experience is related to the quality)

But at the end we have to find out something else what we can show to the Colonel. The problem is that we can measure billions of things from the architecture to the number of open tickets, or the number of exceptions in the logs or Customer Support calls, or what we can imagine. But we would not reach the Quality, only its part, its few aspects. I could imagine a code which has low complexity, well-commented, but poorly designed, or has hundreds open tickets in its issue tracking system, or its UI has bad user experience, and so on...

What I said to my superiors, we cannot produce a number what our Colonel wants, because it is impossible. We have to identify the measurable factors of quality, gather the numbers, create a number from these numbers, and define where is this number comes from, what it means (definitely not the Quality).

We've found these main factors:

• Results of static and dynamic code analysis
• Bugs and feature requests in the tracking system

(hopefully the open tickets could cover the User experience area and the fit of purpose, and anything else what we cannot get from a tool)

We must tell from the very beginning that all project has its own number, and we cannot compare or confront these numbers between projects. And this number itself still does not mean anything only its change in time.

And when we reached this point, I found it interesting: it is feasible, it shows something, it is understandable (mostly, and abstract enough) for the business, and acceptable for the engineering.

Great.

What do you think?

Go to the details in the 2nd part..