Agile trip - [Scrum|Kan]ban

For me the solution is Kanban. Maybe it sounds scary first but you can have your useful scrum meetings (standup, retro, planning) if you like.

With kanban:

• you do not need to change anything at the first day, you do not need anyone else to be changed around you to make things work without being frustrated. You can slowly and effectively change what is needed later.
• you can visualize your actual workflow from the beginning to the end. Usually you do much more than to do, in progress, done. If you have 3 rounds of QA and 2 rounds of code review, and a slow and time consuming release process, you can create columns for them. From that very moment everyone (including you) will see how many states does a ticket need to go through before you can forget it. Because this is what matters, not only the to do-in progress-done...
• you do not need to lie that anyone can pick up a task, you can define what column belongs to what part of the team (well it will mirrored in the WIP points anyway), so everyone can see how the load is balanced. If there are bunch of columns what belongs to one or two guy, you will immediately see it (more importantly show it to others)

I like the fact that you can make your current process visible, and you do not need to change it. You will not have problems with estimating tickets from QA or dev point of view, because it does not matter. You do not need to estimate anything because the time will show you what is your velocity (here it is the lead time), and that is your real velocity. that is the amount of time you need to push a feature to live. You will not need to convert the complexity in yours and other's head to time because it is based on time already.

If you worrying about the planning things, that can be solved easily as well. If the amount of tickets in the ToDo column is lower than X you can do a planning meeting and refill it. If only 5 ticket is ready because the project owner was on holiday you put that 5. It is not an issue at all, you just have to do the next planning earlier. If your team members cannot pick up any task (frontend, backend whatever devs) you do not need to solve it asap, or do weird personal planning meetings upfront. If someone is lack of tasks he can pick up something, or request a ticket from the PO, it is all dynamic. You can also create a column to collect tickets for specific releases, and you will always see how much testing effort is needed for that release. You can even define WIP on those columns to avoid big releases. If you have multiple teams working on the release you can use one board with swim-lanes and handle everything together. If you often forget the end-to-end test of a feature implemented by multiple teams in a period of months, just create a column for it...

if you have too much columns you can create more subset of the boards based on the roles in the teams. QA will see different columns than a dev, a project manager, etc.... 

You can do whatever is needed to make your life easier. The WIP will make sure to not be stuck, it will force you to move tickets forward, and the rest depends on you. You can adapt it to your workspace you will not depend on anything. 

Kanban is more like a tool than a process.

Scrum - agile trip 2

Scrum and me... me and scrum... it is a complicated relationship...My last 7 years was nothing but "move to scrum" "be agile" "save the world" (where world===company).
And scrum is like the cake. It is a lie.

I remember the first time we met. Scrum was like a blond, slender, tall Japanese woman.... interesting but really confusing. Our colleague brought the idea to our office (call him Golden Broom, and our office: The Colony). He had a bunch of cards with different colours, he draw a board (ToDo, In progress, Done) and mentioned the well known meetings. One of the main developers instantly bought more colourful cards where we could wrote our stories, and bugs, and whatevers. It started with one team, we successfully multiplied the amount of meeting hours by approximately 1000, but at least kept the production rate the same (its good and bad at the same time), and increased the frustration (or not... not sure about that). And Scrum, because we thought that is our light saber or I mean live saver, were introduced in every team, one by one, voluntarily (actually it was really voluntary, we really thought that will change our life). But we slowly realized Scrum did not solved any of our issues... but it did change a few things. We told ourselves "at least now we/our job are more visible", what was true, but no one wanted to see us/it.

My relation with Scrum did not change. I have been through a few process migrations now, and I never had the feeling: "wow, this works, Scrum is great". Even if there are problems with the team members, with the company, with the aspect of planets, the main problem is with Scrum. It cannot work. Well, that's not true, it can, but only on paper and in a really small subset of reality. Well maybe it is just me, but I think the process out of the shelve should be able to handle reality without raping it.

What's wrong with Scrum?
1. Its whole concept is just not true. Usually the team does not and cannot have all resources+access+knowledge what they need for their work. So they will have dependencies, and Scrum cannot do anything with that (well you can add an extra column). And it will strongly and randomly influence your velocity (the ultimate value).
2. The team does not work without any connection with the rest of the company (even if the Scrum master pipes all the requests). They occasionally have to help to other people, solve issues, and, BLASPHEMY!!! they have to fix bugs, do releases, because:
3. Scrum just gently forgets about the rest of the lifecycle. Staging testing, releasing, firefighting, hotfixes, service releases... "minor" things what randomly and not randomly influence your velocity again.
4. Scrum has no idea what to do with a non-homogeneous team, like DBA, QA, Dev (frontend, backend). They will be loaded differently during the implementation on an epic, and you cannot plan and utilize them easily (without moving them here and there, or ask them to do other things what is not their profession). But the biggest problem is with the QA-Dev opposition. QA and dev does different things. They cannot be each other substitutes in most cases. What a dev can pick up, a QA cannot, and vice versa.  QA is loaded at the end, they usually have to do testing outside of the sprint (staging, live release testing, etc)... So you will have sprints where the devs are not loaded while the QA is, or the devs could not finish their stories, so the QA could only do a little testing, and they were doing nothing or days (well, you know what I mean). Of course they can always help each-other out (=dev should do more testing) but in reality it does not work. Especially not for longer term.
5. list all your issues here what you get from the fact that the rest of the company need to be able to support your scrum ("process is for the people" not other way around, right?), they need to give you good product owners, the business needs to change their thinking, etc...

So at the end your velocity will be a joke. The number what you can deliver for sure will be much lower compared to the number what you can almost deliver. And this unpredictability will slowly degrade the commitment in the team.

I know, Scrum is not the silver bullet, it is not for any team, but as I see it is only for a team which is in a startup company, with almost every people in it, and they do not have too much users, or they are not on live, so they do not have to deal with bugs, super-important customers, or at least they do not have serious releases process. Where serious means someone has to do something for more than a day. So indeed it is not for all team, but for <0.001%.

And the real problem, that Scrum does help a little. If your organization was a chaos before, without traceability and visibility and predictability (well you wont have it here as well), then introducing Scrum can give you the false hope, that the improvements what you have achieved in the first weeks/months will stay, and all your issues/difficulties will evaporate. But in reality, you will always fail the sprints, so you will plan less and less till it will be ridiculous so you start to increase again, because there is no other way to ease the cognitive dissonance what you have (why we cannot do more if we can do more?). And everyone silently accepts the fact that the sprints are never done. You will often have sprint goals like: "complete the sprint", because the tasks are independent not coherent enough so actually the team is not sprinting to one direction.

At the end Scrum will be nothing else, then a way of organizing meetings and doing estimates, and managing tasks.

Solution comes in the next post.

Agile trip 1

The Agile manifesto - for someone in my age - is nothing but a collection of common sense. It is addressing issues what were issues before I was cloned... oh well lot before I started to work in this industry. I have never met anyone who thinks processes and tools can tell us how to do our jobs. I have never met with a product owner who wanted to see and read the documentation and ignored the software, or ever said: "The feature is awesome, straightforward, but unfortunately not every button is documented....". And even if the customer collaboration was far from perfect, no one ever followed the contract line by line, even if sometimes we should have done it for our own good. But I still often see we do not want to adapt to change. As testers we love to create plans (I don't) and follow them, and ask for detailed requirements what already contains everything so we do not need to use our brains anymore during our job, only if we want to for some reason. And as developers who are so into the details implementation or under the spell of a cool/new/rare/fancy tool/library so they want to use it for sure even if it is just does not make sense anymore. Or doing technology migrations, re-factoring/optimization/etc  for their own fun without adding any value to anyone, and keep doing that even if the whole world change meanwhile. Change is somewhat against our default mindset. We love change, but only planned ones. We love change but only those ones what we start. We hate continuously changing focus, re-planning our tasks, dropping things we love in favor of things we do not know. It seems we forget we are not raising children here.... we create software what can maybe live for weeks even if that piece of code is the best thing we created in our life. We are creating short living phantoms; not statues for eternity.  It is like a shoemaker who either working on one shoe during his life, or does not want to sell any of his creations...

Open space projects - do it yourself

Open space source projects.

I spent a few days with configuring Jenkins JClouds plugin and Openstack. My original goal was/is to make it possible to create our complex (>3--6 machines) system for the smoke test what I plan to run after each commit. Currently I am extremely far from that, I do not even think it is possible in a clean and maintainable way, but what I wanted to achieve in the first phase of this project is to have the magical on-demand jenkins slave feature in our currently sandbox-only Jenkins. I really really thought it will be easy.....

Plug an play did not happen.... it can be because our openstack configuration, can be because JClouds and its Jenkins plugin seems to be mostly used and tested/created/whatever for Amazon Cloud, but it does not work without some changes. I can not really recall all issues but here are some:
1. userdata is optional.... WRONG it will die with a nullpointer exception
2. create jenkins user did not work, we had to create it in the image, it just died somewhere after the init silently....
3. it needs the instance name but it wont brother itself to use the hostname or name or id fields in the metadata it only expects it in the tags/name.... of course it is not there by default for openstack
4. if you add it to all responses (well its python right? eeeeeeeasy) the RuninstanceResponseHandler will die while the DescribeInstancesResponseHandler (I think that was he class name) will work.... basically they handling the same kind of response.
5. you must like floating ips because it will use that even if you do not want to
6. and debugging a remote jenkins master is just pain.... random ports... guesses... weirdness

But is there an issue? I mean maybe we are just too lazy and forget the fact that we can be proactive. Even if I personally do not have the knowledge to understand and fix a tool (well, actually it is not true, I am fking awesome), our company, our comrades, our friends, our family, our kids.... well mostly the company where we work has to acknowledge and allow and support tweaking/fixing/breaking the tools we use. Support it with time or with people.

There is no such a thing as free tool as we all know, but really often we expect from the reality to give us everything cheap+instant, because these tools are so internal and hidden that there is no way to explain to the chief business guru master marketing demigod, or costumer we need some time (where some is greater than the amount of hours what you can work overtime without notice) to fix/do something with them... We lie too much? We does not complain too much? We are just extremely lucky we could make all those releases? Does not really matter which one, the fact is there, we have to spend visible time on them. We cannot just recoil when we found an issue. At the end we have to forge the Death Star and if the hammer is broken we have to find a way to fix it. We have to be honest, but even if no one cares, we have to keep on at Darth Vader if necessary till the problem is solved or we are dead.

So at the end the first phase was done, Jenkins slave was started, we had to spend some time on debugging, some other on finding solutions, some more on applying solutions, some more on more debugging, some more on drinking more coffee, but at the end it will work (or not), and we learnt something (or not). And we can tell to our boss, and the costumer and the sergeant: software development is not magic... we are just plain old craftsmen

Automation

I really love the autobots, the automatron, the automobile, the automauto, basically everything which starts with auto and end with suffer and pain....

Automation.... our silver bullet, life saver, job saver, mind saver.... I have made and work with a few automation framework (internal), and one thing always happens if you are not paying attention to it... it will be a heavy, massive piece of sh*t... why? you are a test engineer, you are not creating unit tests in these frameworks, so it is pretty sure you will have to implement some business logic (simplified form), you will execute complex, more complex scenarios, because your mind is critical you will do negative, unhappy, sad, destructive tests. You are working in an "agile" environment, trying to follow the code generators (alias developers) support the release, act as an internal customer support (and all these things are fun). You want to deliver the "tests" asap, they want you to deliver the "tests" asap. You try to avoid headaches so you wont revisit any story only if you have to, so you try to be fast, fast, fast....

And meanwhile everyone forgets your/our automation framework is an application as well. It needs maintaining, it must be structured, no code duplication, easy to learn, easy change, but most likely some of them will be true for you, who created it, but not for anyone else...

You end up with a regression test suite running for 4 weeks, because it was not designed to run it in parallel, or to have any kind of run optimization. But to add it you have to change everything. The code is completely unbalanced, some parts are good, others are so crappy you just do not want to open it. And the tests are te same.... some using an old part of the framework, in the others you tried out something else, the latest are working in a different way. It has so many tricks (workarounds) even yourself start to lose track, but you cannot do anything, because the code generators still generating code, you want to follow them...

Solution:
make your framework, your tests from the very first time as they are mission critical applications. Because they are. You cannot be negligent, the execution and the report coming out of your tests will be the most important thing for your team (if not, do not do it at all).

Sonar cSs plUgin

I know, I know... no post for a year and 2 today... the mini-mes (aka young clones) are not here, I have plenty of time...

It is just pure self-praise but I just wanted to tell you that the sonar-css-plugin is now available (not officially released yet) in the sonar forge. So if you have CSS files under your pillow, now you can parse it with its awesome AST parser, and check against brilliant, super-intelligent, cosmological, incomperable (https://github.com/CSSLint/csslint) rule set.

You can find it here: https://github.com/SonarCommunity/sonar-css

I am open to suggestions, ideas, requirements, pull requests, donations, followers, worshipers

Un-unit-test-ablility

I truly and deeply regret my long silence. Because of various reasons I had to move to another battleship. Slightly smaller but we can react faster to any action of the Rebellion. But it is not that important. What important is, I have a new codebase to work with....

it is in Java, and it is completely up to date with the trends. Which is something really new to me. The last time I was involved in a java upgrade for a project, was from 1.4 to 1.6 when the 1.7 was about to coming out. It is using a lot of java 8 features (lambdas, Consumers, soon Nashorn,  etc) it is a fully concurrent, event based, scalable thing (not sure what else, but I can recall these words)....

so (or not so, just and)...

which is nice, on one hand, it looks funny, especially after erlang, and some javascript. I also created my first lambda expression and I felt a strange prickling over my scalp and over my arms, it was not a bad feeling... But I am a tester, (at least my superior shouts it all the time) and the project lacks unit tests, I decided to lets do some unit testing, and show them (=devs) it is so easy to do:

1. there was a itsy-bitsy feature, what was about getting some number from another system, store it during the session, and when it ends dump it to somewhere. I thought, if something is unit testable than thats it. we can check what happens if the other system sends a negative value, or a 0 (what should not happen) or a number what is less than the previous (should not happen as well)... its easy. I prepared myself that I have to use mockito (not mochito, but at the end I should have used that) to mock out the other system, and everything I do not really need,  and I can show them, how to do it.
well...
the whole system is concurrent (whatever it means) and asynchronous, so the class which had that (private) method I wanted to test had lambdas all over the place (and inner classes). It was passing complex classes, the Runnables were triggered from various places, and I just get lost in it... just to test that method I either had to mock half of Corellia, or had to create instances and start up almost everything. So I just said: There is no way I gonna unit test this....

2. but I am stubborn so I decided to give another try for unit testing in this app. There was a "unit" test, what was testing a small class which was parsing some buffer/stream/whatever coming from the network. I was like: oh yeah, here I can truly mock the shit away, simulate the stream what this class gets, from its field, and remove the dependency of an external running system during this test (which is an issue with this test). I tried to not mock the whole world around it, just what I need, but I failed again... mocking the class what stores the buffer was not enough because I had to call a method in my class to read the buffer from the field, but it also used some other fields from the mocked class which was not initialized because it was just mocked... but if I had not mocked it I could not changed the data it holds....

so at the end I realized, with my knowledge (I am somewhat sure this is the main issue), in reasonable amount of time it is impossible to unit test anything in this application (well thats not true, but at least find something new what is not unit tested already). All these Runnables lying everywhere and triggered from various places when a given event happened somewhere, I felt like I wanted to directly test an anonymous function deeply nested into other anonymous functions in a javascript code.... just no way....

so is it me or it is the world?