When to use waterfall, when agile?

Software projects failed a lot in the past. They failed to deliver the value for the business, were too late or ways out of budget. The selected process method was usually the scapegoat for the failure with agile methods being the answer to any question in software development. But as usually in live, it’s not black or white. The selection of the right software process method depends on the surrounding of the project. I gathered some industry input and combined it to reflect the current thinking regarding agile software development methods vs. traditional methods.

The adapted Stacey matrix

Adapted Stacey Matrix for technology / software development environment

The original stacey matrix supports decision making processes suggesting appropriate management actions and defines four areas: simple, complicated, complex and chaotic. The suggested actions depend heavily on the context of the decision making.

The dimensions of the adapted matrix: HOW and WHAT

The x-axis of the adapted matrix deals with the HOW. If the team knows the technology well and has used it many times before, we’re on the left. Otherwise, if the technology is completely new to the team we’re on the right of the dimension. The y-axis positions the WHAT. On the bottom of the axis, the stakeholder of the project all agree on the goals and have the same understanding of the expected outcome. On top it’s the opposite, no agreed requirements and no alignment on expectations. The individual mix of the project points to a certain area with a process model suggestion in the adapted matrix.

Waterfall …

Waterfall is a traditional project management method with sequential steps and no iterations. Massive upfront planning is done before any implementation work starts. If all goals and steps are clear, waterfall produces consistent results in a predictable and repeatable way. The clearly defined tasks lead to an optimized sequencing and optimal resource allocation. Waterfall optimizes resources and return on invest if cause and effects are clear to anybody in the project team.

… vs. Agile

Agile stands for SCRUM, Kanban and LEAN methods with flexibility, quick response and constantly changing environments in mind. They start quicker with smaller scope for the current increment with the scope being like a rolling window. Uncertainty of the projects’ goals needs quick adjustment and adaptation during the whole execution. Only close and frequent collaboration with all team members make agile projects successful. If causes and effects aren’t clear, agile works in small steps towards a value-generating and broadly accepted result.

Simple to chaotic – from “known knowns” to “unknowables”

Simple = easily knowable, the known knowns

Projects in the simple zone unveil very few surprises, decisions are fact- or evidence-based, advancement occurs in orderly, sequential steps and the WHAT is clear to anybody. Any size projects with clear activities and repeatable results fits in this category. It has been done multiple times before and best practices exist as benchmarks. The process is simple and could be handled in a check-list style.

Going forward in a simple, fully predictable project means reducing it to the maximum to make the single pieces easier to understand. Examples of simple: recipes, tasks on an assembly line, checklist based work.

Complicated = not simple but still knowable, the known unknowns

The complicated zone segments into socially and politically complicated and technically complicated. Complicated means less simple but still somewhat predictable.

In Social/political complicated environments people can not agree on the purpose of the project and the expectation on results is not clear. Requirements are conflicting amongst the diverse stakeholder which could be resolved with waterfall to get clearance on WHY before WHAT before HOW. On the other hand applying agile techniques could help convincing stakeholders to agree on already achieved results and smoothing the further requirements discussion. The project team needs to pay special attention on getting early agreement between stakeholders in place.

In technically complicated contexts it’s clear on WHY and WHAT to achieve. Still, the HOW is not clear. An agile iterative approach helps getting feedback from the project team on the achievements making adaptations possible.

Going forward in a complicated project means as well reducing it to the maximum to make the single pieces easier to understand. Technically complicated is e.g. using a specific technology for the first time. Political/social complicated is e.g. if the relation between cause and effect are not clear enough or conflicting opinions amongst various stakeholder exist.

Complex = not fully knowable but reasonably predictable, the unknown unknowns

The complexity zone stands for high risk and uncertainty and requires a high feedback frequency. Neither requirements nor the execution are clear. Holistic defined process methods don’t work any longer. The context asks for a more explorative approach with transparency, frequent inspection and adaptation. SCRUM as a process method in the toolbox of the agile mindset is the method of choice. It increases transparency with small iterations and frequent check-points allowing cheap adaptations. The team planning is the start point for each new iteration and allows immediate feedback from stakeholders to the teams to adapt the next iteration.

Complexity can not be reduced, some understanding can be achieved and complexity can not be planned, it simply grows. A good example of a complex project is software development in general. The requirements are rarely fully defined right at the beginning and it’s seldom clear which architectural solutions are superior to others.

Chaotic = neither knowable nor predictable, the unknowables

In chaotic zone requirements and execution path are both undefined and the risk is high. Kanban as the most flexible project management method is the tool of choice. With no structure like sprints and the only focus on work in progress (WIP) Kanban focuses on continuous delivering results to allow further modifications in direction and backlog items.

The goal is to move from chaotic towards complex by dividing the problems. The principle “Act, Sense and Respond” helps navigate towards the zone of complexity.

Sources – from where I learned:

Innovation and organizations – Hackathon vs. RocketLab

Innovation is usually part of agile product development methods. Sometimes, however, agile methods just replace other methods. SCRUM replaces Waterfall, KANBAN formalizes previously unordered work. Obviously, the innovation dilemma remains still open. Where comes the creativity from? The ideas? Where to test those hypotheses which are not part of the daily routine?

The hackathon as innovation tool

Some organizations run hackathons once or multiple times a year. We did and are doing this as well. We organized already 6 hackathons in the past. Once yearly. Did we see the innovation boost? Well, yes – and no.

How do we organize a hackathon?

A hackathon at gutefrage.net is a timeboxed activity (usually 2,5 days) and surrounded by a lot of social activities. We cook, we bake, we experience Virtual Reality, we do some board games, we play the football table and have a good time and fun. The whole company participates usually and is excited to validate hypotheses which are usually not part of the product development. There are no limits from a topic perspective. Teams organize themselves via a democratic voting exercise right at the beginning. People pitch their ideas and convince other people to become part of this specific project group.

What’s the typical outcome?

During the hackathons at gutefrage.net one out of five ideas launch during the hackathon. This one idea is production ready and creates value right from the launch. The other ideas typically proof aspects, create prototypes of various qualities, cover maximum the best-case implementation and still need an investment of 80% to be ready. The hackathon is a great team building event, it’s great for the morale, the culture. The hackathon drives people’s motivation and frustrates them if their project doesn’t make it into the finals.

What issues do we see?

The hackathon validates some hypotheses, some not and the question remains open what to do with all the started work? Will we follow some traces? Will we just abandon the work? Needless to say – after the 2,5 days hackathon there waits daily business in form of agile software development work. At gutefrage.net we promised to launch the winning idea and typically abandon the remaining work. We found that’s not the most efficient way to drive innovation.


The RocketLab – our way to innovate

As a learning organization we drew some conclusions from the hackathon experience. Mixed teams with participants from all relevant areas worked very well. The one project going live was a real push for team and company motivation. The others weren’t as good for morale. Thinking about this for a while we came up with a slightly different format – the RocketLab.

What’s different between RocketLab and the hackathon?

The RocketLab stands for outcomes and can potentially solve any topic: Product, Technology or others of cross-discipline relevance. At day one of the RocketLab there is one specific hypothesis the team focuses on. The team exclusively works for a defined period solely on solving this one issue. No distraction, just 100% focus. The team contains all disciplines to solve the issue on hand. They are all committed to create the best solution possible within the given time budget. It’s a team of maker, not talker, not theorists, no visionaries. The hackathon is broad and unspecific by nature, the RocketLab has a given goal to accomplish with the solution delegated 100% to the team.

How do we organize a RocketLab?

It’s typically either Product or Technology bringing up a specific hypothesis (or a technical complex problem to solve). A short discussion determines the amount of time we’re willing to spend on finding a solution – usually 3 to 5 days. The organizers invite people to participate in the RocketLab and the Lab kicks off. It’s never the whole company, only few people but interdisciplinary.

The initial task after kick-off is an intense planning session. The organizers introduce the hypothesis to a greater detail and the team sets goals – together with metrics. Right afterwards with a clear goal in mind and a good understanding of the metrics solution ideation starts. Ideally, the team ends this activity with a solid set of tasks for each team-member.

The RocketLab needs 100% dedicated team members – no excuses – and sits co-located in a special meeting room.

What’s the typical outcome?

The expected outcome of the RocketLab is a solution for the hypothesis from the beginning. The solution is live, up and running. If the team was not able to solve it 100% they have a clear understanding of the remaining efforts and a thorough plan. The plan is then executed in regular agile development work. One hypothesis, one implementation, one proof. The RocketLab is an efficient and effective tool to work concentratedly on a hypothesis – goal-oriented but very intense.

What issues do we see?

We did around 10 RocketLabs for very different topics. Very concrete, technical topics up to very abstract conceptual work. The results were sometimes simply spot on, other times needed further perfection during daily work. In essence, the RocketLab is a tool which borrows aspects from the hackathon but is more effective and efficient. It simply works for us and produced some very surprising solutions.

We still see some issues with the spill-over effect of the RocketLab – but that’s a minor problem. Only 20% of the Labs experienced the spill over.

graphics
rocket – used under creative commons license (CC BY 4.0), non-modified
hackathon logo “The Hacking Dead”: © by gutefrage.net

Toolbox, Best Practice, Tools, HowTo’s – agile practices and how to apply them

The internet holds quite a lot of information. Here’s my favorite collection of tools, toolboxes, methods, best practices and howto’s from various fields of application. Most of them have a tight coupling to agile software development, agile organization development, product development and cross those fields.

Following a list of links and a short description of what to find on the site.

Open Practice Library, https://openpracticelibrary.com/

The site contains methods and tools around product Discovery and Delivery practices. The methods are collected by the community and serve to inspire seeking minds to test them in various situations. The methods are positioned in 4 main areas: Discovery, Delivery, Options Pivot and the Foundation.

Discovery contains methods like:

Options Pivot holds tools like:

Delivery contains these tools:

Foundation hold something like these:

Ideo – Design Kit, http://www.designkit.org/methods

This is a collection of tools around Inspiration, Ideation and Implementation. The kit is provided by IDEO.org.

Inspiration

Ideation

Implementation

Other Tool Sets