When to use waterfall, when agile?

Software projects failed a lot in the past. They failed to deliver the value for the business, were too late or ways out of budget. The selected process method was usually the scapegoat for the failure with agile methods being the answer to any question in software development. But as usually in live, it’s not black or white. The selection of the right software process method depends on the surrounding of the project. I gathered some industry input and combined it to reflect the current thinking regarding agile software development methods vs. traditional methods.

The adapted Stacey matrix

Adapted Stacey Matrix for technology / software development environment

The original stacey matrix supports decision making processes suggesting appropriate management actions and defines four areas: simple, complicated, complex and chaotic. The suggested actions depend heavily on the context of the decision making.

The dimensions of the adapted matrix: HOW and WHAT

The x-axis of the adapted matrix deals with the HOW. If the team knows the technology well and has used it many times before, we’re on the left. Otherwise, if the technology is completely new to the team we’re on the right of the dimension. The y-axis positions the WHAT. On the bottom of the axis, the stakeholder of the project all agree on the goals and have the same understanding of the expected outcome. On top it’s the opposite, no agreed requirements and no alignment on expectations. The individual mix of the project points to a certain area with a process model suggestion in the adapted matrix.

Waterfall …

Waterfall is a traditional project management method with sequential steps and no iterations. Massive upfront planning is done before any implementation work starts. If all goals and steps are clear, waterfall produces consistent results in a predictable and repeatable way. The clearly defined tasks lead to an optimized sequencing and optimal resource allocation. Waterfall optimizes resources and return on invest if cause and effects are clear to anybody in the project team.

… vs. Agile

Agile stands for SCRUM, Kanban and LEAN methods with flexibility, quick response and constantly changing environments in mind. They start quicker with smaller scope for the current increment with the scope being like a rolling window. Uncertainty of the projects’ goals needs quick adjustment and adaptation during the whole execution. Only close and frequent collaboration with all team members make agile projects successful. If causes and effects aren’t clear, agile works in small steps towards a value-generating and broadly accepted result.

Simple to chaotic – from “known knowns” to “unknowables”

Simple = easily knowable, the known knowns

Projects in the simple zone unveil very few surprises, decisions are fact- or evidence-based, advancement occurs in orderly, sequential steps and the WHAT is clear to anybody. Any size projects with clear activities and repeatable results fits in this category. It has been done multiple times before and best practices exist as benchmarks. The process is simple and could be handled in a check-list style.

Going forward in a simple, fully predictable project means reducing it to the maximum to make the single pieces easier to understand. Examples of simple: recipes, tasks on an assembly line, checklist based work.

Complicated = not simple but still knowable, the known unknowns

The complicated zone segments into socially and politically complicated and technically complicated. Complicated means less simple but still somewhat predictable.

In Social/political complicated environments people can not agree on the purpose of the project and the expectation on results is not clear. Requirements are conflicting amongst the diverse stakeholder which could be resolved with waterfall to get clearance on WHY before WHAT before HOW. On the other hand applying agile techniques could help convincing stakeholders to agree on already achieved results and smoothing the further requirements discussion. The project team needs to pay special attention on getting early agreement between stakeholders in place.

In technically complicated contexts it’s clear on WHY and WHAT to achieve. Still, the HOW is not clear. An agile iterative approach helps getting feedback from the project team on the achievements making adaptations possible.

Going forward in a complicated project means as well reducing it to the maximum to make the single pieces easier to understand. Technically complicated is e.g. using a specific technology for the first time. Political/social complicated is e.g. if the relation between cause and effect are not clear enough or conflicting opinions amongst various stakeholder exist.

Complex = not fully knowable but reasonably predictable, the unknown unknowns

The complexity zone stands for high risk and uncertainty and requires a high feedback frequency. Neither requirements nor the execution are clear. Holistic defined process methods don’t work any longer. The context asks for a more explorative approach with transparency, frequent inspection and adaptation. SCRUM as a process method in the toolbox of the agile mindset is the method of choice. It increases transparency with small iterations and frequent check-points allowing cheap adaptations. The team planning is the start point for each new iteration and allows immediate feedback from stakeholders to the teams to adapt the next iteration.

Complexity can not be reduced, some understanding can be achieved and complexity can not be planned, it simply grows. A good example of a complex project is software development in general. The requirements are rarely fully defined right at the beginning and it’s seldom clear which architectural solutions are superior to others.

Chaotic = neither knowable nor predictable, the unknowables

In chaotic zone requirements and execution path are both undefined and the risk is high. Kanban as the most flexible project management method is the tool of choice. With no structure like sprints and the only focus on work in progress (WIP) Kanban focuses on continuous delivering results to allow further modifications in direction and backlog items.

The goal is to move from chaotic towards complex by dividing the problems. The principle “Act, Sense and Respond” helps navigate towards the zone of complexity.

Sources – from where I learned:

Design Thinking in a nutshell – what is it and what’s in for us?

What is Design Thinking?

Design Thinking puts one person group in focus – the user. All activities in Design Thinking circulate around making the user’s life better. Design Thinking is an activity where all stakeholder take part. Not only designers, product owners or developer are part of the group, marketing, community management, finance and even legal should be involved in this very early stage of the product development cycle. Why should they? It’s a matter of understanding and communication. All participants of the Design Thinking process are part of goal setting, reasoning and the detailed planning. They need to share the vision behind the product. Time is wisely spend in the beginning to smooth the following implementation steps. Involvement of participants? 100%. No e-mails, no meetings, just the user and their pain points.

Other names for Design Thinking are “Design Sprint” (Google) or “Iteration 0” (infoq) or “Design DOING”. Design Thinking is a great planning tool to let all people understand what is build when and with what purpose. The process consists of a set of methods typically executed directly before the typical lean and agile development cycles start. It defines what to build and to communicate this purpose amongst the participants in an efficient, effective and fun way.

What’s the result of Design Thinking?

The result of the Design Thinking activity is a verified prototype, common understanding of what to achieve and a plan on how to proceed building the Minimal Viable Product and which features to add after going live. On top of that comes the certainty to build a set of functions that user’s really want and need – not a feature an executive has fallen in love with. Further definition of Design Thinking: IDEO, interaction design foundation or wikipedia.

What’s the purpose of Design Thinking?

  • Customer centricity uncover real user needs
  • Boost communication and understanding bring business and technology together – let the whole team understand what to build and for what
  • Create business value create and test solutions for these user needs
  • Be better and faster build better products and bring them to market sooner
  • Be focused and measurable extract clear goals from real customer needs
  • Get prioritization right feature prioritization gets a lot easier (no “pet” features)

Phases of Design Thinking

Design Thinking is a fun exercise, fast paced with the goal to create a concrete problem definition and an implementation plan for the most promising solution.

5 Phases of Design Thinking

Starting with an idea of a user problem in mind the user observation phase results in a better understanding of the real user pain points. The team identifies the root pain points of the users and document them in an experience map. Creation of multiple solution ideas means to apply ideation techniques with the clear goal in mind of not implementing the first-thought-about solution. The paper prototype challenges the solution idea from ideation with the lowest possible effort and tries to resolve the user’s pain points. The implementation planning starts right after the finalization of the paper prototype. The result of the Design Thinking process is an implementation plan.

Implementation Guideline – Design Thinking in 10 Steps

1 – Identify user pain points

Go out of the building, watch, observe and interview real users experiencing the problems to solve by the solution you’re about to build. Watch at least 8 individuals! User’s verbal feedback usually contains their thinking of solutions, not their problems. The understanding of the team prior to the user observation is significantly different from it after watching and talking to users.

2 – Show the customer journey on an experience map

The experience map shows the current product experience (if any) and the user pain points. Organize notes from the team’s observation phase on sticky notes on a wall. All members of the Design Thinking team take part in the observation phase and make their own notes. The experience map shows the flow from the beginning (left) to the end (right) of the user journey.

Experience Map

From top to bottom the experience map shows the separate steps (above in blue) in the user journey. Below the user journey steps yellow notes represent the observations from the team members. Themes / cluster – represented in green – create an umbrella for the various observations. Additionally, emotional states – e.g. with smileys – enrich the overall journey and make it more visual.

3 – Extract and prioritize pain points

At this stage the experience map contains contains a lot of observations from various people. The next step in the process is to extract the pain points to focus on during the remainder of the process. “Dot-voting” is the tool of choice here. Each participants gets 4 to 5 dots. The biggest pain points receive the dots. Dot-voting happens all at once – at the same time. Observations may receive none, one or many dots from each participant. Counting the dots identifies the most important pain points.

4 – Goal and metrics setting

Having the ordered list of pain points extracted, the next step is to convert them into goals and make the progress measurable. Setting the goals can result in either quantitative or qualitative metrics. If possible assign real numbers to the goals. Orientation criteria for setting the goals can be:

  • efficiency (e.g. time spent on task)
  • effectiveness (e.g. reduction of # of errors)
  • satisfaction (customers’ happiness with the solution)

5 – Create example people – personas

Instead of developing a product for anybody – and hence most likely nobody – define real people, personas. Be specific and narrow user segments down. The persona should include the name, attributes, goals, concerns, quotes and other emotional elements. Ideally extract only 2 to 3 personas either individually or in small groups. Share the results afterwards with the rest of the group.

6 – Ideation – create ideas to solve pain points

Now it’s time to think about multiple solutions to address the user’s paint points. Don’t jump to solutions – don’t build the first “obvious” solution. Most likely there are more clever approaches available. All participants take part in the solution creation phase. Individuals create ideas time-boxed (e.g. 15 minutes). Afterwards they share them with the broader audience. There are no stupid ideas – it’s important to listen carefully to all ideas – some of them will stipulate further thinking. After the individual session a group session to detail the most promising ideas follows. Involving all team members creates shared ownership for the solution.

7 – Reality check

During ideation the goal is to create a lot of ideas. Are they all feasible? Can they really solve the issue? How does this look in reality? Since the ideas are in early stages we need to check them with reality. Place the ideas on storyboards. The storyboards show the interactions of the users with the idea. They are a sequence of scribbles which fit easily on a sheet of paper and focus on the positive user path. Let all edge cases, negative paths, recovery steps aside. Just focus on the positive user experience path associated with the idea. Reality check happens as an individual task or in smaller groups.

8 – UI Paper Prototype

The reality check sketches the user flow on a very high level – only focusing on the positive path and keeping intermediate screens aside. With the Paper Prototype the user flow should be as realistic as possible. Now the team members need to think through intermediate screens and pop-ups, edge interactions with the user and error cases as well. They transform the high-level storyboard an user interface. Sticky notes represent the UI elements – so the elements can change easily.

9 – Usability testing

Now it’s time to get real feedback from real users on the ideas developed. Any feedback is welcome and influences the prototype iterations. Feedback is usually more open since the prototype signals “Hey, we’re still in early phases. Don’t hesitate to give honest feedback!”. For the overall product changes at this stage are ways cheaper than rework done in later phases. Identify around 5 test participants – ideally based on the persona descriptions used so far in the design process. The storyboard serves as the base for tasks for the participants. The UI needs to change as an answer to the actions of the participants. Do so by exchanging notes or other elements of the UI manually. During the Paper Prototype testing give the participants no hints. The overarching goal is to identify gaps or errors in the user flow and collect feedback to improve the prototype.

10 – Implementation planning

The last step to finish the Design Thinking process is to come up with an agreed-upon implementation and launch plan. The plan is incremental in nature, delivers business value as early as possible and involves all disciplines. Implementation planning usually results in a story map which reflects the desired state of future interactions. Jeff Patton explains story maps in great detail in his book “User Story Mapping”. Here are only some important aspects.

Story Map

What’s the story map?

The story map is a one page explanation of the big picture and shows details of the planned product or feature. It includes a release strategy, describes iterations around the minimum viable solution and identifies areas for additional research. First of all list all stages of the user journey as they use the product. This is usually similar or equal to the blue sticky notes produced in the experience map. Below, in green, put the concepts of the UI which help to fulfill the step in the user journey. Yellow notes hold capabilities of the product extracted from the paper prototype. Take a note on the sticky notes of the expected outcome the capability delivers to the user. The yellow notes usually translate into implementation epics in agile development. Put pink notes on capabilities marking additional research need.

With the finished story map, you have a quite comprehensive view of what needs to be build for a perfect solution. To further mitigate risk of failure you want to start with the least minimum effort in development – identify the minimum viable product feature set. Now, it’s time to sort the yellow sticky notes below the UI areas according to their criticality to the product success. Sort them into these categories having the question in mind: “How necessary for users is this capability to fulfill their task?”:

  • critical: absolutely necessary for user to get tasks completed
  • commercially acceptable: adding these features will allow commercial success
  • later: add capability later
  • nice-to-have: capability which can be added in later phases if time allows
Story Map with prioritized capabilities

When finished with the prioritization, you have your iteration planning finished. The top lane includes all capabilities needed for the minimal viable product. Next lane adds functions to enable commercial success. Later lanes store further functions to add to the product to become a full solution.

Find some videos on lynda.com by Chris Nodder on the specifics of each step and the applied technique (need a Lynda.com subscription).

Design Thinking – Timeline

Typically Design Thinking activities fit into a 5 days week. You might want to organize them like this:

Design Thinking – week template

Where to start to speed up your IT environment – Here are 5 areas to look after.

Anil Cheriyan shared his thoughts on where to focus to create a fast and better working IT environment in the financial services industry – to speed up the organization (see: https://enterprisersproject.com/article/2017/6/suntrust-cios-formula-speed-relies-cloud-devops). In his post he mentioned 5 areas you should look after to break with old habits and start creating a fast pacing environment.

Anil Cheriyan is Director/Deputy Commissioner, Technology Transformation Services for the U.S. Federal Government. Previously, he was managing partner of Phase IV Ventures, a consulting and advisory firm.

Cloud

Two important aspects associated with the terminology “Cloud”. First it’s important to understand the implications of the various cloud strategies (ranging from private cloud over hybrid constructs towards public clouds). Get your strategy clear on which areas to host where. Criteria to look at are: business value provided, business continuity, resilience, security. Second aspect is the organization. Get your people involved. They need to participate in the strategy definition. They will execute them actually. No time for information hiding and bimodal IT infrastructures.

Modular architecture

Getting towards a modular architecture introduces flexibility in decisions, eliminates bottle necks and allows a decentralized governance. Today’s architectures are still monoliths or more advanced SOA stacks or somewhere in between. A more modular architecture exposes API’s via micro services. This architecture allows distributed ownership models. Complex is actually the implementation of these architecture rewrites. A lot of business related activities and the re-architecture work is a hard effort to combine.

DevOps

DevOps is finally all about the mindset of people and the break-up of silo-ed organizations. People need to learn and understand the importance of collaboration and trust. This sounds simple, turns out to be a heavy change project. Anil started pilot projects and introduced the true DevOps mindset and collaboration through success cases. It’s not about adopting rules and processes from the DevOps movement “by the books” – it’s about training your talent to work closer together.

Agile development

Agile development in software development is quite wide spread and commonly used. The acceptance over waterfall models is – where appropriate – high. Issues occur if the agile software development processes get surrounded by traditional waterfall-oriented functions – control functions. The most challenging part is to get agility into release management, deployment and integration testing.

Design thinking

Most important aspect of design thinking is the customer centricity. Understanding the real problems of the user to be solved is at the core of the approach. Not hunting the 100% perfect solution with all nice and “useful” features. Going for the most valuable solution, ship it fast. This requires heavy re-thinking within the organization. It’s more about talent and collaboration models. Important is to get people together with a thorough understanding of the industry and processes to help solving the customer’s pain points.

Architecture alternatives for rendering a web site

There’s a great overview of technologies available from Google comparing the different architecture options to render a web site. Jason Miller and Addy Osmani present options from SSR (server-side rendering) over various mixed models to complete CSR (client-side rendering). They describe the pros and cons of the various approaches and give hints on what to use in which situation. A great read!

https://developers.google.com/web/updates/2019/02/rendering-on-the-web

Rendering options

  • SSR: Server-Side Rendering – production of HTML is done on the server
  • CSR: Client-Side Rendering – creation of HTML is done on the client, usually using the DOM
  • Rehydration: Using a JavaScript based client app to show the server-rendered HTML – mixed with the DOM tree and associated data
  • Prerendering: generation of HTML is done during build time

Performance acronyms

  • TTFB: Time to First Byte – time between clicking a link and the first bit of content
  • FP: First Paint – time until any pixel gets visible to the user
  • FCP: First Contentful Paint – time until the requested content (article, body, ..) becomes visible
  • TTI: Time To Interactive – time until a page becomes interactive

Jason and Addy wrap their great article up with an overview of the options. Since it’s presented under Creative Commons Attriubtion 3.0 License I decided to reproduce it here for further reference.

Toolbox, Best Practice, Tools, HowTo’s – agile practices and how to apply them

The internet holds quite a lot of information. Here’s my favorite collection of tools, toolboxes, methods, best practices and howto’s from various fields of application. Most of them have a tight coupling to agile software development, agile organization development, product development and cross those fields.

Following a list of links and a short description of what to find on the site.

Open Practice Library, https://openpracticelibrary.com/

The site contains methods and tools around product Discovery and Delivery practices. The methods are collected by the community and serve to inspire seeking minds to test them in various situations. The methods are positioned in 4 main areas: Discovery, Delivery, Options Pivot and the Foundation.

Discovery contains methods like:

Options Pivot holds tools like:

Delivery contains these tools:

Foundation hold something like these:

Ideo – Design Kit, http://www.designkit.org/methods

This is a collection of tools around Inspiration, Ideation and Implementation. The kit is provided by IDEO.org.

Inspiration

Ideation

Implementation

Other Tool Sets

Realistic case study on agile development at large scale

A Practical Approach to Large-Scale Agile Development” by Gary Gruver, Mike Young and Pat Fulghum is a real-world example on how scaling of agile software development really works in a huge software producing organisation.

A Practical Approach to Large-Scale Agile Development

The authors describe in an easy-to-read language the journey of the HP firmware organisation starting in 2008 and taking around 3 years with a clear goal in mind: “10x developer productivity improvement”.

In the beginning they were stuck with a waterfall planning process with a huge planning organisation and not being able to move in software development as fast as business expected. A quick summary of activities showed that in the beginning the organization was spending 25% of developer time in planning sessions to plan the next years’ releases. Only 5% was spent on innovation. Nowadays, after acomplishing the 10x goal, 40% of developer’s time is spent on innovation.

The book highlights the relevance of a right mix of agile technologies with a good approach in software architecture and organisational measures to form a successful team of people striving for common goals. A fascinating read!

Most striving is the unemotional view on agile and how to apply it. They purposefully decided not to have self-organizing teams. So, agile is broken? Can’t be applied in such an environment? Not at all! The authors give good reason for not applying all agile patterns from the books – and it is working.

Agile at LEGO

Quote

I am interested in agile software development methods, LEAN management styles and the impact on a company’s organization. All these attributes are usually associated with software development companies – not hardware companies or toy companies.

I got a “must-read” recommendation for the book “brick by brick” by David Robertson. "Brick by Brick" by David RobertsonThe book proved to be a real “must-read”. Especially the portion where David Robertson describes the turnaround of the company. The methods applied to an almost bancrupt company reminded me to techniques nowadays well-known in software development teams. I mailed David Robertson and asked for permission to cite a portion of his book (page 170, Brick by Brick, David Robertson):

Before the rise of Bionicle, the LEGO Group’s product teams were soloed from one another and toys were for the most part developed sequentially: designers mocked up the models and then threw their creations over a metaphorical wall to the engineers, who prepared the prototypes for manufacture and then kicked them over to the marketers, and so on down the line. Rarely would one team venture onto another team’s turf to offer a suggestion or ask for feedback. If all went well, the team’s product would hit the market in two or three years.

The Bionicle team’s six-month deadlines forced a different way of working, one that was less sequential than parallel, and highly collaborative. Once the outline for the next chapter of the Bionicle saga was roughed out, the different functional groups would work side by side in real time, swapping ideas, critiquing models, and always pushing to simultaneously nail the deadline and build a better Bionicle.

“We had a massive project team,” recalled Farshtey. “It wasn’t just the creative people; it was also people from Advance and from marketing, sales, events, PR – all different parts of the company, all helping to steer the franchise.”

Because the marketing group worked directly with designers, Bionicle’s advertising campaign felt connected to the product. Promotional posters for Bionicle’s first-year run had to look and feel of movie posters, precisely because the toy featured the powerful visuals and narrative sweep of an epic film. “We wanted more communication in the product and more product in the communication,” said Faber. “That meant the marketing group needed to be involved at the very start of product development, so the story flowed out through the product. We wanted the product almost to tell the story by itself.”

“We had a kind of triangle, where the marketing, the story, and the product had to move ahead together,” he continued. “None of those could be the spearhead. Each needed to support and inspire the other.”

That single page really got to a point. I’m relentlessly telling that the core of the agile movement is about focus and collaboration (see Agile Defined). Usually, I refer to these values in software development environments, organizations and teams. But LEGO – a brick & mortar company – used the same principles in 2001 to turnaround their company. That was really mind-boggling news to me.

After doing some research on this topic, I found a SMB here in Germany working with agile principles: HEMA Bandsägetechnik. This (german) article talks about their experience.

 

Customer & Service Provider interface – the End of the Story

Customer & Service Provider interface … the End of the Story 

With the start of the-then-new job in October 2015 I started writing about my experience – and especially the shortfalls – in our customer and service provider relationship. It turned out to become a triptych. This is the final – the wrap-up.

To catch up you might want to read the first post: “Customer & Service Provider – Epic fails” and the second in this series: “Customer & Service Provider – silver lining on the horizon“.

You might ask yourself … why is that the end of the story? Improvements in processes usually don’t have an end. Well, let’s see.

The Modified Setting

In the meantime we upgraded from 2,5 software developer to even 3,5 software developers. Ambition behind is …

You might guess it? …

Come on …

YES! Right – on the first attempt.

Speed up the delivery of features. Our senior management is delighted by the achievements so far – however is far from being satisfied with the results, the outcome. Still, there is great belief that more people build more results. True! BUT we’re still having the issue of not well-defined requirements and still having long periods of wait for answers from our business leads.

On a daily basis we’re still in communication mode. Twice a week our project manager and the project manager of the service provider meet to exchange progress and start discussion on new topics. Again, communication proves to be a gold nugget!

The Now Situation

Missing product management – It became so obvious, so non-neglectable, so crystal-clear. Reason behind the low througput of our development cycle is not the lack of software developers – that’s a symptomatic solution. No, root cause is: we’re lacking a professional approach to manage our products – product management is missing. Usually, product management’s major duty is to define requirements on their products and harmonize the various needs and demands from all involved parties – customer, departments, management.

If we just had some people doing exactly this job. World would be much clearer!

  • Prioritization would be easier – since the product management would clearly include management in the overall prioritization process.
  • Definition of requirements would be clearer – only well-defined stories were handed over to our service provider.
  • Decision points would be clearer – product management is the definition point for detailed questions and processes.
  • Communication would be simpler – it’s simply clear whom to ask for guidance.
  • Development would be faster – no impediments any more … why should it be slow now?

In my last post, I raised a lot of points:

  • Multiple stakeholders talking to the service provider
  • No clear communication of priorities, requirements and goals
  • Fundamental changes in design after going live
  • Bad requirement quality
  • Silo thinking
  • Low quality – not well thought-out
  • No consistent prioritization

All of them – I really believe – all of them could be solved with the implementation of an organizational change – introduce product management.

So, you might ask yourself why is this the end of the story? Well, that’s an easy one. Despite my continued effort to convince my senior management about the absolute need to improve the organization by introducing a product management person (yes, one person!), I failed. “Too expensive”. I personally decided to quit.

Lesson learned?

I’m still convinced we changed a lot for the good in the customer / service provider interface. Agile patterns did a great difference for all of us. Furthermore, the service provider did invest in agile coaching (and architecture consultancy as well … not all is gold …) and showed great willingness to cooperate – to improve.

Go on, improve your interface. It’s well worth the effort! Ah, and don’t forget to secure your senior management’s backing in early stages 🙂

Customer & Service Provider – silver lining on the horizon

Customer & Service Provider interface … epic fails and how to overcome them

In a recent post I was talking about an interesting situation regarding the collaboration with our service provider (see http://www.agile-minds.com/customer-service-provider-epic-fails/). After some weeks I’d like to take a chance to draw some conclusions and report back some successes and learnings. I definitely see some silver lining on the horizon regarding the collaboration part – but also discovered some real black holes within our organisation. It’s always the same: process insufficiencies will always show up – latest – in the IT department …

The Modified Setting

Now in the development partnership with our service provider we pay for 2,5 software developers + 0,75 project management person. This means we added significant ressources to the aspect of project planning, communication and management.

Our project management person and the pendant at the service provider really met twice a week to discuss progress, issues and need for further planning on specific tasks. In addition to that it’s me and the general manager of the service provider meeting almost weekly to discuss issues and problems. This approach helped to improve the communication a lot. We’re now in a continuous dialog. Understanding of each other has raised enormously. A great step!

The New Situation

We are still not happy with the outcome – but the reasons have changed dramatically. Previously, we as an organization were not able to articulate our needs and get this information over to our service provider. We changed our communication paradigm and introduced the face-to-face communicaion twice a week. Now, the service provider has a ways better view of our requirements and is eager to start implementation. However, a lot of our stories lack quality from a subject matter view. How did we come to this point?

Multiple stakeholders talking to the service provider – We only have one person talking about new requirements – our project manager. He’s responsible to collect requirements in-house and communicate them back to our service provider. This works quite good. We’ve setup internal bi-weekly calls to collect and prioritize requirements. We’ve had our first quarterly face-to-face meeting where we started to build and organize the company backlog.

No clear communication of priorities, requirements and goals – Flag it as solved with the previous point.

Fundamental changes in design after going live – This is still an ongoing topic. But it’s a question of educating our internal people. Since we’ve one single person doing the communication towards the service provider it’s one of his duties to explain people that if there’s a change in requirements after going live we’ll treat this change as a new requirement – and this needs to get prioritized and implemented. Education.

The New Symptoms

No progress over all – Nope. Having progress. Or eager to go on. But …

Bad requirement quality – This was not transparent so far. But it came up when we introduced the new way of working. It became clearer and clearer that our subject matter experts – the stakeholder – didn’t exactly know what they wanted – not even at the point when they were talking to the service provider. Or they knew what they wanted – but didn’t consider the other business units in the design phase.

The Idea … did work out so far

The two aspects I’ve decided to implement

  • Setting up a stable and reliable single face of contact on working level
  • Introducing basic agile principles to get to stable requirements

worked out smoothly so far. As usual the introduction of agile aspects increased transparency of involved processes and especially the quality of the different artifacts a lot. And new issues turned up.

Quality of Requirements

As said previously the introduction of agile principles – well, I’d say of new communication patterns – produced process insufficiencies of our own organization.

Silo thinking – our business is separated in different business units. The different units have their own targets and try hard to reach them. There is no competition between the units – but everybody takes care of their own business. Two of the business units eventually use the same instance of the student information system. Especially between those units we have a lot of cases where e.g. the layout of the exam certificates differs, rule sets for course acceptance differ, specializations of courses differ and so on. Small differences – which might be needed … or not – which have a big, big impact on implementation capacities. It’s a difference to create different rule sets for courses – double the implementation work. And why? Ah, the business stakeholder didn’t have time to discuss their individual needs and combine them into mutual needs.

Low quality – not well thought-out – Another point we found is the quality and depth of the various business requirements. A lot of our business requirements arrive at our project manager at a stage where it says “We need a proper implementation of …”. Well, what does “proper” mean? Many times we experience developers from the service provider receiving last-minute changes – still. That’s one of the focus points for future improvements.

No consistent prioritization – Our exec management wants to review the implementation prioritization. Fair enough. We’ve agreed on having a call/meeting every second month. In this call we’ll run through the company backlog and listen carefully to what exec management sees as top priorities. In parallel we have our working calls with the business stakeholder every second week. Together, I believe, we’ll arrive at a prioritization which puts most important topics first.

My biggest learning out of this so far?

No matter how big the issue – start introducing agile patterns. Here especially the communication, company backlog and the face-to-face meeting helped a lot to unveil the real issues behind the “bad customer / service provider” interface.

The nice part about the agile pattern is the transparency. It automatically points you to the real pain points – through transparency.

Stay tuned and read on – to be continued.

Customer & Service Provider – Epic fails

Customer & Service Provider interface … epic fails and how to overcome them

Just recently I decided to change positions – moving away from a CTO role with a whole development department attached towards a CTO role with all software development tasks being outsourced to a service provider / product company. And I felt like being in a time-warp … not into the bright new and shiny future – no … directly into middle age – the darkest version of it.

See why – and how I plan to maneuver all of us out of this.

The Setting

The product company is very specialized on delivering a specific product aiming at university campus management software. We’re paying 2,5 developers to work full-time on our product branch – we call it a development partnership.

The product company develops in PHP, does deployments with puppet – however utilizes a full tomcat environment and PostgreSQL as database. They claim to work in an agile manner … and utilize JIRA (yeah!). Peeking into their JIRA unveils a per-person-association-to-tasks (well …). Visiting the development team produces clean white walls, no empty bottles or cans around, no architecture diagram scribbles, and no nothing. Suspect environment for full-heart agile developers. But let’s see.

The Situation

We are not happy with the performance of the service provider. Why? They don’t deliver on time, in quality and make promises all the time. Hmmm. After few visits and talking in-house and the service provider the situation became clearer to me.

Multiple stakeholders talking to the service provider – We had multiple parties of our business units talking to various people at the service provider. All of us spoke with different voices and everything was super-important. In the end not the feature with highest business impact made it live – it was the one of the person shouting loudest…

No clear communication of priorities, requirements and goals – In our discussions with the various people at the service provider we were not able to communicate a clear order of priorities nor were the requirements defined – not talking about documented or clear – and goals of the implementation tasks were sometimes not even clear to our internal stakeholders.

Fundamental changes in design after going live – I think that’s a real classic: The whole implementation is done, the product change is live and the stakeholder suddenly realizes that the color coding of a logo needed to change. So, the stakeholders didn’t even realize that changes in the live product are the most costly changes – compared to those done on powerpoint or photoshop level right in the beginning.

The Symptoms

No progress over all – Having not communicated a goal, a strategy or anything with a guiding function it’s no wonder there is no felt progress in the cooperation with our service provider. Sure, everybody was giving his/her best and we were moving. But nobody was satisfied at all. We didn’t get what we wanted and the service provider felt bashed at every move they made.

Communication went really personal – I personally attended the at-the-time two-weekly phone call with our service provider. We had the CEO, 3 business unit leads, one project manager and another 3 subject matter experts on the call – together with one of the managing directors of the service provider. As usually, I learnt afterwards, the call ended in real bad communication on a personal level. There was no common sense, every word was political, no commitment to anything – it was protect my a** everywhere.

The Idea

This needed to end. So, after my first week I had a meeting with the board of the service provider. Board means two people – the founder and investors. I presented them with an idea to improve the situation. At the core of the idea there were two main principles:

  • Setting up a stable and reliable single face of contact on working level
  • Introducing basic agile principles to get to stable requirements

They bought into this approach and we agreed to set this up.

Stable and reliable single face of contact on working level – I made my project manager to spend 80% of his time to manage the interface to our service provider. The service provider put 40% of a person into this task.

The two of them meet twice a week. Idea behind is to get closer communication flowing. The sooner we know about problems, the faster we can solve them. Anything not clear? Any issue to solve? Too less resources? Whatever. On Tuesday, they meet to discuss the current stories in implementation and issues. Furthermore, they discuss future stories to get feedback on expected implementation issues and too less specific requirements. On Friday, they meet to discuss the same topics and in addition I and the managing director meet to discuss and decide on topics to be resolved.

Internally, we made clear that it’s my person and the project manager who speaks to the service provider – nobody else. NOBODY! This is the only way to remove distracting conversations on non-important topics and puzzling requirements. Not to talk about setting other priorities on tasks. This all happens when the internal stakeholder have direct access to the service provider. Now, the project manager collects the user stories and requirements. We meet on a frequent basis – at least once a week, usually every day – and exchange our knowledge on user stories.

Introducing basic agile principles to get to stable requirements – Obviously, we had XLS-sheets of various formats and contents. All of them full with requirements and sorted to various priorities. We kindly removed all of them and started our Company Backlog.

We agreed with the service provider to have sprints of 2 weeks. The service provider still does releases on a 4 week basis – but we work in 2 week sprints. This allows us to communicate delays of features in terms of 2 weeks instead of month. “Well, it didn’t make it into this sprint – we will put it into the next sprint. This starts in 2 weeks.”

The Company Backlog is now organized in JIRA. We attach all relevant information to the user stories and hand them over to our service provider. Unfortunately, we couldn’t agree on using the same JIRA instance. Initially, I thought to use a Company Backlog view where we organize the next sprint which is then handed over to development and they start associating tasks to the user stories.

The daily standup is a bi-weekly get-together of the relevant people and starts to work out pretty good. During these meetings the two of them do a kind of backlog grooming and discuss the next stories to be implemented.

So, to summarize. Agile elements are:

  • The Company Backlog
  • The Product Owner (on our side – still named project manager)
  • The Scrum Master (on service provider side – also named project manager)
  • Sprints of two weeks duration
  • Backlog grooming
  • Planning 1

Does it work? Well, I’m sure it will be an improvement after all. Is it optimal? Let’s see!