Abstracting abstract abstractions

Even on Dagobah there is no such a thing: enough rain.

Abstractions are good. We can simplify things based on the way how we want to use it. The last time when I was on a QA conference on our one and only Death Star in the Nether sector, there were a whole presentation about how can we create a good framework with creating layers of abstractions.

Lets take an example:
    You have to test a backend user management system, which has a REST API(or SOAP, or json over http or basically whatever). When you have to test it you will create an abstraction for sure which maps your tests to actual http calls somehow. and hopefully you won't make http calls from your test methods every time... This is layer 1.

    After a while, when you get bored to set up the same object over and over again when you want to log in with a test user, or register, or do something you will decide to create another abstraction with util methods for those commonly used actions or group of actions you have to perform over and over again. This is layer 2.

   After a few months or after adding much more tests or after covering many many new features you realize even with these util methods sometimes it is painful (in terms of lines of code, or in terms of readability / maintainability) to set up complex preconditions. So you decide you need another layer. But it is not obvious what.

    Maybe you already have a class file which represents your test users. So it can hold basic or not basic info like: first name, last name, dob, email, password, number of dogs, IQ etc... And maybe you will start to wonder: if I have a list (or something like a list) of attributes what a user can have, why do not I have a list of actions what a user can perform, or can take. You can have things like: login, register, send message to someone, get banned, fly to the moon, get fired, get attacked by Jedis etc...
    One of the best way to list things in java is to create an enum. Another benefit is, you can define an abstract method, what every enum must implement, something like: doAction(TestUser). So in you enum thing, you can list your actions, and implement the required api calls to achieve that action (use things from previous layer). And in your test user object you can do a method like: doActions(MyOwnActions... action) and you can list those things what you want to perform with your test user. This is layer 3. and it is really handy:
user.doActions(REGISTER, LOGIN, GO_TO_MOON, KILL_A_JEDI, EAT_BAKLAVA);

    So we have 3 layers for a simple api testing which abstracts the http calls to a level where you can just list what you want to do and your test user will be in that state. But what happens if someone testing the GUI for example needs your test user thingy. You can tell him to do what you did, so before the tests create a test user instance and set it up with the required actions and so on.
It means they have to change or have the same kind of things on multiple place (of course depends on, maybe you have to keep your test class hierarchy flat, and you do not need users in every test). So maybe in that framework it can be ugly to set up the test users everywhere, and you want to provide a nice way to just get what you want.
    You already have a static abstraction about the actions what your test users can perform, so you can easily create dependency injections.
    Well I know that this example is not that common, but in my case we cannot just inject the dependency with guice or something... here, in the tests the test user depends on runtime parameters, and you need a given and often various state of the user.
    What can you do? You can create an annotation like: @TUser(actions={Action1, Action3}) and if you use testng you can override the IMethodInterceptor so you can create a test user into the annotated field based on some parameters (from the test instance, or runtime things) you decide to use. So you have abstracted the test user initialization, and this is layer 4.

...well, when I implemented this I thought there are at least 8 level of abstractions .. it's only 4, but the message is clear, if things are getting complicated and you have lot of duplication do not afraid to take a step back, look it from a new point of view, and put those things what you need to another layer.

...sorry for the half broken language, my brain is half off....
 

Preconditions

In a few hours every day I have to test a Death Star-like application. I know there is nothing special about it, but this DS is a naughty one. It has billions of settings, and because it is a shared environment it often happens that something has been changed in the cms shortly before testing, and the QA does not know about it. So when we run the tests we always have hundreds of false positive results. (it is better than a false negative but it is annoying)

So on a quiet day I've decided to create a Precondition framework.
It has two main targets:

1. Make possible to add preconditions to every test we have, in a simple and common format.
2. Make possible to evaluate those preconditions before test run and skip the tests with a message if the precondition fails.

The framework have to be somewhat independent from the test framework (at least the first part), and could be independent from the actual level and type of testing (unit, integration, UI, API...).

So how does it is look like? Like a Thermozodium esakii.

The marking part is annotation based. You can annotate your test methods and your test classes with one @Preconditions which may have more (at least one) @Precondition. The @Precondition has the following parameters: id = the unique id (in class level) of this precondition, these can be used for linking. type = any value from the CondiotionType enum (to collect the precondition types in one place). args = the required arguments for the evaluation. skip = for skipping the evaluation for a given value of a given test parameter (see later). description = a short description which will be in the message of the SkipException.
You can mark your class fields with a @TestParam annotation which can be used for a simple value checking of the args value of its @Precondition (see examples below).

The evaluation is triggered by a TestNG listener class which implements the IInvokedMethodListener, in its beforeInvokation method gathers the required annotations and calls the required class for the evaluation and throws and exception if it did not met.

So you can define any type of conditions you only have to implement it in the *.conditions package (or elsewhere) and create an enum value for it, and add it to the beforeInvokation method of the listener (its a huge else-if... no better solution yet).

Here is a few annotation examples (using TestNG):

 public class MyGreatTest{  
   /**
   * The test field what should have a specified value
   */
   @TestParameter(conditionID="testParamCondition")  
   public String myTestParameter;  

   /**
   * The field gets its value here
   */
   @BeforeMethod  
   public void setUp(){  
     //... do something  
     myTestParameter = someMethod();  
   }  

   /**
   * A precondition which describes the check of a simple value of a parameter (of course the implementation has to be created before)
   */
   @Preconditions(@Precondition(type=ConditionTypes.parameterCheck,args={"YouCanDoIt"},id="testParamCondition",description="To check the field has the required value"))  
   @Test  
   public void firstTest(){ 

   }  

   /**
   * This example is one of our most common one. Based on the values of the parameters of the test method we check something in the CMS and if it has a NULL set skip the test. For some reason we can skip the evaluation if we define the skip argument. It contains the value and the position of the test parameter we want to check. So in this case, we define: if the first parameter (which is firstParam in this case) has the value 'Atlantis' skip the checking of this condition.
   */
   @Preconditions(@Precondition(type=ConditionType.checkSomethingInACMSByXpath,args={"/universe//planet//region[@='Aqua']"},skip={"1","Atlantis"},description="This test requires an Aqua type region, except when the first parameter has the value: Atlantis"))
   @Test
   public void secondTest(String firstParam, Integer secondParam){

   }

 }  

Why is it good for us? You can check the precondition in the before method and skip the test if it does not match... If you have 1000+ tests it is useful to part your tests from precondition checking, and you don't have to change the code on a lot of places, and you can easily skip the evaluation if you remove the Listener for the actual run.