When to use waterfall, when agile?

Software projects failed a lot in the past. They failed to deliver the value for the business, were too late or ways out of budget. The selected process method was usually the scapegoat for the failure with agile methods being the answer to any question in software development. But as usually in live, it’s not black or white. The selection of the right software process method depends on the surrounding of the project. I gathered some industry input and combined it to reflect the current thinking regarding agile software development methods vs. traditional methods.

The adapted Stacey matrix

Adapted Stacey Matrix for technology / software development environment

The original stacey matrix supports decision making processes suggesting appropriate management actions and defines four areas: simple, complicated, complex and chaotic. The suggested actions depend heavily on the context of the decision making.

The dimensions of the adapted matrix: HOW and WHAT

The x-axis of the adapted matrix deals with the HOW. If the team knows the technology well and has used it many times before, we’re on the left. Otherwise, if the technology is completely new to the team we’re on the right of the dimension. The y-axis positions the WHAT. On the bottom of the axis, the stakeholder of the project all agree on the goals and have the same understanding of the expected outcome. On top it’s the opposite, no agreed requirements and no alignment on expectations. The individual mix of the project points to a certain area with a process model suggestion in the adapted matrix.

Waterfall …

Waterfall is a traditional project management method with sequential steps and no iterations. Massive upfront planning is done before any implementation work starts. If all goals and steps are clear, waterfall produces consistent results in a predictable and repeatable way. The clearly defined tasks lead to an optimized sequencing and optimal resource allocation. Waterfall optimizes resources and return on invest if cause and effects are clear to anybody in the project team.

… vs. Agile

Agile stands for SCRUM, Kanban and LEAN methods with flexibility, quick response and constantly changing environments in mind. They start quicker with smaller scope for the current increment with the scope being like a rolling window. Uncertainty of the projects’ goals needs quick adjustment and adaptation during the whole execution. Only close and frequent collaboration with all team members make agile projects successful. If causes and effects aren’t clear, agile works in small steps towards a value-generating and broadly accepted result.

Simple to chaotic – from “known knowns” to “unknowables”

Simple = easily knowable, the known knowns

Projects in the simple zone unveil very few surprises, decisions are fact- or evidence-based, advancement occurs in orderly, sequential steps and the WHAT is clear to anybody. Any size projects with clear activities and repeatable results fits in this category. It has been done multiple times before and best practices exist as benchmarks. The process is simple and could be handled in a check-list style.

Going forward in a simple, fully predictable project means reducing it to the maximum to make the single pieces easier to understand. Examples of simple: recipes, tasks on an assembly line, checklist based work.

Complicated = not simple but still knowable, the known unknowns

The complicated zone segments into socially and politically complicated and technically complicated. Complicated means less simple but still somewhat predictable.

In Social/political complicated environments people can not agree on the purpose of the project and the expectation on results is not clear. Requirements are conflicting amongst the diverse stakeholder which could be resolved with waterfall to get clearance on WHY before WHAT before HOW. On the other hand applying agile techniques could help convincing stakeholders to agree on already achieved results and smoothing the further requirements discussion. The project team needs to pay special attention on getting early agreement between stakeholders in place.

In technically complicated contexts it’s clear on WHY and WHAT to achieve. Still, the HOW is not clear. An agile iterative approach helps getting feedback from the project team on the achievements making adaptations possible.

Going forward in a complicated project means as well reducing it to the maximum to make the single pieces easier to understand. Technically complicated is e.g. using a specific technology for the first time. Political/social complicated is e.g. if the relation between cause and effect are not clear enough or conflicting opinions amongst various stakeholder exist.

Complex = not fully knowable but reasonably predictable, the unknown unknowns

The complexity zone stands for high risk and uncertainty and requires a high feedback frequency. Neither requirements nor the execution are clear. Holistic defined process methods don’t work any longer. The context asks for a more explorative approach with transparency, frequent inspection and adaptation. SCRUM as a process method in the toolbox of the agile mindset is the method of choice. It increases transparency with small iterations and frequent check-points allowing cheap adaptations. The team planning is the start point for each new iteration and allows immediate feedback from stakeholders to the teams to adapt the next iteration.

Complexity can not be reduced, some understanding can be achieved and complexity can not be planned, it simply grows. A good example of a complex project is software development in general. The requirements are rarely fully defined right at the beginning and it’s seldom clear which architectural solutions are superior to others.

Chaotic = neither knowable nor predictable, the unknowables

In chaotic zone requirements and execution path are both undefined and the risk is high. Kanban as the most flexible project management method is the tool of choice. With no structure like sprints and the only focus on work in progress (WIP) Kanban focuses on continuous delivering results to allow further modifications in direction and backlog items.

The goal is to move from chaotic towards complex by dividing the problems. The principle “Act, Sense and Respond” helps navigate towards the zone of complexity.

Sources – from where I learned:

Agile at LEGO

Quote

I am interested in agile software development methods, LEAN management styles and the impact on a company’s organization. All these attributes are usually associated with software development companies – not hardware companies or toy companies.

I got a “must-read” recommendation for the book “brick by brick” by David Robertson. "Brick by Brick" by David RobertsonThe book proved to be a real “must-read”. Especially the portion where David Robertson describes the turnaround of the company. The methods applied to an almost bancrupt company reminded me to techniques nowadays well-known in software development teams. I mailed David Robertson and asked for permission to cite a portion of his book (page 170, Brick by Brick, David Robertson):

Before the rise of Bionicle, the LEGO Group’s product teams were soloed from one another and toys were for the most part developed sequentially: designers mocked up the models and then threw their creations over a metaphorical wall to the engineers, who prepared the prototypes for manufacture and then kicked them over to the marketers, and so on down the line. Rarely would one team venture onto another team’s turf to offer a suggestion or ask for feedback. If all went well, the team’s product would hit the market in two or three years.

The Bionicle team’s six-month deadlines forced a different way of working, one that was less sequential than parallel, and highly collaborative. Once the outline for the next chapter of the Bionicle saga was roughed out, the different functional groups would work side by side in real time, swapping ideas, critiquing models, and always pushing to simultaneously nail the deadline and build a better Bionicle.

“We had a massive project team,” recalled Farshtey. “It wasn’t just the creative people; it was also people from Advance and from marketing, sales, events, PR – all different parts of the company, all helping to steer the franchise.”

Because the marketing group worked directly with designers, Bionicle’s advertising campaign felt connected to the product. Promotional posters for Bionicle’s first-year run had to look and feel of movie posters, precisely because the toy featured the powerful visuals and narrative sweep of an epic film. “We wanted more communication in the product and more product in the communication,” said Faber. “That meant the marketing group needed to be involved at the very start of product development, so the story flowed out through the product. We wanted the product almost to tell the story by itself.”

“We had a kind of triangle, where the marketing, the story, and the product had to move ahead together,” he continued. “None of those could be the spearhead. Each needed to support and inspire the other.”

That single page really got to a point. I’m relentlessly telling that the core of the agile movement is about focus and collaboration (see Agile Defined). Usually, I refer to these values in software development environments, organizations and teams. But LEGO – a brick & mortar company – used the same principles in 2001 to turnaround their company. That was really mind-boggling news to me.

After doing some research on this topic, I found a SMB here in Germany working with agile principles: HEMA Bandsägetechnik. This (german) article talks about their experience.

 

Agile – criteria to look for in an Agile business

In February 2013, I made an attempt to define Agile on my blog. I ended in a quite high-level definition of Agile:

Agile is a collection of values and principles that encourage a certain type of behaviour: focus on value generation and collaboration.”

Just recently, I found an interesting blog by Rouan Wilsenach working with ThoughtWorks where he went one level deeper and explained “Four Attributes of an Agile Business

1. Feedback – “This is what my customer wants.”

In my definition this relates to focus on value generation. Companies need to put the customer in the center of all their efforts. Rouan refers to an interesting list / toolbox on experience design. Only if organizations take this attribute serious they will finally be successful applying principles of agility.

2. A responsive team – “Yes. We can do that.”

In my definition this very much relates to values and principles supporting collaboration. Responsiveness comes only with co-location. In my experience, true collaboration over a distance can only be achieved within a trusted environment. People need to trust and respect each other to become a distributed team. If they don’t know each other of follow slightly different goals, it’s very likely to fail. So, the team spirit can be created most easily when forming inter-disciplinary teams which are located ideally in the same room. Rouan refers to Conway’s law (comparison of code structures with organizational patterns)

3. A responsive code base – “It’s ready. Shall we release it?”

For me, this again a perfect example of value generation. Have a feature ready? Why not shipping it immediately? Great entry for principles like “Zero Bug Policy” or “Continuous Deployment”.

Furthermore, a responsive code base is a clean code base. People love to work on such a code base. It’s almost bug free, easy to read, simple to understand, documented. People are proud of what they’ve developed, created.

4. Continuous direction – “What do we do next?”

Again, this point goes as value generation. Why employing a bunch of people working hard on software development, building great products if there is no direction behind, no strategy?

Clearly one of the major tasks of managers in an agile environment is to set the strategy. Product people break this strategy down into epics and stories. So, what to do next? Pick the next story from the prioritized backlog.

So, for me Rouan picked some really great criteria and attributes for an agile business. They’re in line with my previous definition (which I like) and take it to a less abstract level.

Agile defined.

How is Agile defined? What does it mean?

In recent discussion within my company and also in discussion with other people, I recognized that people use the term Agile a lot. It is also obvious that people partly have a thoroughly different understanding of the meaning of Agile.

When reading about the topic in the internet, following some people on twitter and reading books about agility and similar topics it becomes apparent that Agile has arrived at mass-movement. It is no longer well-understood and sharply defined. The term is more or less a buzzword. Think of “SOA”, “Test Driven Development”, “High Availability” or “Big Data”. All of them arrived at the buzzword-level. Anyway, that’s hard to change.

Agile defined

Looking up Agile at a dictionary it returns:

agile, adjective

  1. quick and well-coordinated in movement
  2. active; lively: an agile person.
  3. marked by an ability to think quickly; mentally acute or aware

So, we’re talking about an adjective – a closer description of the state of something. Agile doesn’t stand on its own. It refers to something. Interesting. But what does it refer?

Agile in organizations – defined

Agile is not a framework

In the context of agile organizations the term very commonly gets confused with other agile things. A lot of people refer to their organization as agile since they introduced SCRUM or Kanban as their software development frameworks. These people confuse Agile with frameworks with comparable core values and motivations. In SCRUM the core values are commitment, openness, focus, respect and courage. However, SCRUM as a framework focusses only on a certain aspect of the overall Agile movement.

Agile is not a methodology

Furthermore, some people think Agile is a methodology. If you follow well-known process steps, applying always the same pattern to certain situations you apply a certain methodology to arrive at a goal. But Agile is not a collection of best-practices, a rule-set and you’re fine.

Agile is not a goal

Others look at their organization with the sole ambition to become Agile. But there is no state you can arrive at and claim – “Now I’m Agile”. Agile is the path, not the goal.

So what is a definition for Agile in organization context?

Agile defined

Read the great blog post from Jeff Patton about Agile development is more culture than process. Also a great source of insights is the Agile Manifesto.