Product Development Process – notes on “How to Create Tech Products Customers Love” – #8/11

Product Development Process -notes on "How to Create Tech Products Customers Love"

Product Development Process

The new mantra in product development is:

“Move Fast and Don’t Break Things.”

In 2014, Facebook changed its mantra for developers from “Move Fast and Break Things” to “Move Fast with Stable Infra”. Only moving fast doesn’t bring the business benefit home. At a certain tipping point the engineering organization introduces more “broken things” than they’re able to fix in the long run – speed of the organization stalls or declines.

The new mantra relies on two cornerstone principles Release Quickly and Often and Release with Confidence. Releasing quickly and often leads to small and quick iteration cycles. These foster quicker learning and fuel innovation. Releasing with confidence keeps damage away from your brand, your revenue streams, your customer base and your employees. Confidence refers to accurate software, reliable and well-performing releases, scalable software and released software without privacy or security concerns.

A good example of a well performing product organization is Google Search. They run a total of 15 product teams. They work on over 2.000 product ideas – no product optimizations – in Discovery with less than 500 built in Delivery. The ratio is 4:1 – 4 ideas in Discovery, 1 get build. They aim for 10.000 ideas in Discovery with less than 500 being built. See “Rigorous testing” from Google for further information. Google mentions in this article they did 595.429 Search quality tests, 44.155 Side-by-side experiments, 15.096 LIve traffic experiments resulting in 3.234 Launches – in 2018 alone!

The industry benchmark in well operated digital product development organizations is to kill 75% of ideas in Discovery. If not a minimum of 50% of the ideas are killed then it’s Design, not Discovery and a clear signal of a malfunctioning organization.

Issues of Conventional Agile

“Agile is all about building and delivering software, but it says almost nothing about how to come up with a valuable product backlog.”

Marty Cagan

Agile software development helps a lot, but doesn’t say anything about building a valuable backlog. It is completely blind on the value side, answering the what should be built. Marty discusses a solution to this issue – dual-track agile – on his blog.

Issues of Conventional Agile:

  • Thanks to agile, teams move faster. But agile doesn’t provide teams with a valuable goal.
  • Moving fast without a goal leads to feature chasing. The goal of agile is to produce more output, not necessarily more business results or more value. This is especially tricky with mature products where removing features might provide more value to the business and customer (use A/B testing to understand the value). Also in mobile products more features most often fail – they distract more since there is too less inventory – and finally don’t have an impact.
  • Engineers are most time only coders and don’t contribute further more to the overall value generation process.
  • In agile it’s very hard to predict any dates and make commitments since it’s based on user stories only.
  • UX Design suffers because there is simply no place for UX in agile. The sprint iteration doesn’t reserve time slots for UX to work as part of the iteration. If engineering is ready to go UX should be ready as well but usually start right now to work as well.
  • The architecture of the system suffers as well. Fast moving teams produce more technical debt. Without really knowing good ways to solve an issue, suboptimal long-term solutions are implemented and increase the need to repair architectural decisions.
  • The rest of the company – marketing, sales, customer support … are brought in too late into the process.

Salesforce rolled out agile by the book about 15 years ago. As a consequence the whole UX & Design team threatened to quit because conventional agile didn’t include UX & Design at all. So, after some discussion Salesforce included UX & Design in their agile working model and pivoted from the Conventional Agile approach.

Continuous Discovery and Delivery – Dual Track Agile

“Discovery, by definition, means you don’t know the answer when you start.”

Ed Catmull, Co-founder of Pixar
Parallel Product Discovery and Delivery - Dual-Track Agile
Parallel Product Discovery and Delivery – Dual-Track Agile

“Dual-Track Agile” emphasizes the parallelism of Product Discovery and Product Delivery. They happen all the time in parallel. The Discovery track is all about fast learning, building a validated product backlog. Delivery is all about creating software that could be released with confidence, shippable software. The Discovery track is led by Product, Delivery by Engineering.

Product Discovery provides missionaries with purpose. In Discovery, the key risks are addressed and meaningful answers to these questions give guidance to the team. The given answers are all validated, there is enough evidence proving it will work.

  • Will they buy it? (Value)
  • Can they use it? (Usability)
  • Can we build it? (Feasibility)
  • Can our stakeholders support it? (Viability)
  • Bonus question: Should we build it? (Ethics)

When all critical questions have good and validated answers, describe what to build in Delivery. Therefore, use e.g. JIRA stories with prototype and UX design (prototype-as-spec).

It is, however, not needed to validate all risks all the time in discovery before putting anything into the backlog. The trio of people (PM, PD, TL) does the risk assessment (takes 5 minutes talking). Pick and choose the techniques to validate if needed. Don’t be religious about the process.

Example: At ebay, one base assumption is: create a safe place for transactions. This assumption is the root cause for the overall reputation system within ebay.

Discovery and prioritized backlogs

Don’t prioritize your backlog – simply try thousands of ideas!

Marty Cagan

Follow only those ideas in favor of the overall business objectives. Don’t spend too much time thinking about them, just try them. One iteration in Discovery means: One new idea or another approach to an older idea.

Put the ideas that work into the product backlog. If the product backlog is empty switch to “feed-the-beast” mode and allow the teams to work on e.g. technical debt or other meaningful tasks. If the product backlog is too full – means more than 2 weeks of work – you have a rotten backlog. Here, ideas will no longer be valid or well memorized if too much time has gone before going into Delivery.

Good example for a Product Discovery phase and how fast they were collecting evidence and pivoting: BMC Business Modell Competition – First Place Winner: Owlet (https://www.youtube.com/watch?v=f-8v_RgwGe0)

This blog post is part of a series. It summarizes my personal notes of the workshop held by Marty Cagan “How to Create Tech Products Customers Love” from 5th to 6th of June in 2019 in San Francisco.

Product Discovery Principles – notes on “How to Create Tech Products Customers Love” – #9/11

Product Discovery Principles

“The most important thing is to now what you can’t know.”

Marc Andreessen

1) We know we can’t count on our customers to tell us what to build

Avoid focus groups. Customers don’t tell you what to build because they simply don’t know what is possible with latest technologies in place. Furthermore, they don’t know what they actually want until they see it – hence use prototypes during your customer feedback sessions. Customer-facing product teams go 3 hours per week to see the customers. “Customer inspired – customer enabled”.

The iPhone wouldn’t happen with focus groups. Skype hired Nokia, Motorola and Apple people and they compared the different approaches to innovative products. It took 3 ½ years to build the iPhone. Nokia and Motorola were doing focus groups when Apple started the iPhone. Palm has just released the Treo with Touch Screen and people didn’t like it. The focus group gave clear indication to skip the touch screen. So, at the time, Motorola and Nokia decided not to build phones based on touch screens. Apple hat a vision – the iPhone vision – “Build a touch screen powered phone.”

2) The most important thing is to establish value

Value for customers, creating value for customers. Typical product roadmaps line up features. The underlying assumption of feature roadmaps is that there’s value in the feature and business viability is granted. But that is typically not true.

3) We recognize that engineering is hard, but the user experience is often even more difficult, and more critical to success

This is especially true at B2C companies – 50 engineers and 2 visual designers don’t work out. The experience and value perceptions created at the customer is vital – UX is more critical to success than technology.

4) We recognize that functionality, design, and technology are inherently intertwined

5) We expect that most of our ideas won’t work, and those that do will require several iterations.

Many iterations never make it beyond us – the team; they’re simply stopped internally in early stages.

How many iterations should one follow before skipping the idea?

  • Depends on the importance of task
  • Ask yourself: “Are you still learning with every iteration?”
  • Change the approach to the problem (e.g. churn)
  • Timebox: 2 days for one approach

Avoid the “fall-in-love”-pitfall with design: use less than 2 days for design before result is shown to users.

Example: Ceramic class separated into two groups with separate goals. The first group should build one high quality pot. The other group is instructed to build as many pots as possible – it’s only the weight that counts. The second group won the quality challenge because of practice! The first group wanted to produce the “perfect” pot but failed with just 1 mediocre pot.

6) We must validate our ideas with actual customers

Go out of the building, talk to real customers. It’s valuable to start with your team mates, colleagues, people working inside your company – but finally, you need to get the opinion of your real customers.

7) We validate our ideas as quickly and cheaply as possible

Possible even in hardware: 15 iterations per week. Google Glass team build an engagement model prototype and improvised a futuristic user interface with simple technologies. They quickly learned about a key problem: shoulder sourness and skipped the key assumption for success.

8) We use both quantitative and qualitative techniques

Quantitative: What’s happening?

Qualitative: Why is it happening?

Example: Etsy switched to endless scrolling from pagination. The A/B-Test showed people buying less. The qualitative analysis showed: there were simply too many cool things to buy and people couldn’t decide – paradox of choice

9) We must validate both technical feasibility and business viability in product discovery, not after

Include engineers ways before planning. They need to be involved and see what to build before sprint planning.

10) It’s all about shared learning

Shared learning happens all the time in a co-located team. Discussions over coffee, exchange of opinions at the work-desk, discussions are all around. Furthermore, allow engineers 30 minutes playtime with the prototypes each day to let them express their concerns and start having good ideas on how this could be build.

With a distributed team in place, schedule a 30 minutes meeting every day to allow the exchange of ideas, to give engineers air time with the prototype and help them understand the ideas behind.

Big companies run innovation labs for product discovery. But good ideas never materialize due to the separation of Discovery and Delivery.

This blog post is part of a series. It summarizes my personal notes of the workshop held by Marty Cagan “How to Create Tech Products Customers Love” from 5th to 6th of June in 2019 in San Francisco.

Product Discovery Techniques – notes on “How to Create Tech Products Customers Love” – #10/11

Product Discovery Techniques - notes on "How to Create Tech Products Customers Love"

Product Discovery Techniques

Don’t use these discovery techniques for bug fixes or optimization. They’re meant for real product discovery work, identifying and foremost validating big new product ideas.

1) FRAMING

Framing is the activity where the problem space is defined and the relevance of the problem at hand gets better understood. Do not spend too much time in framing.

See Marty’s blog for a more thorough description: https://svpg.com/assessing-product-opportunities/.

Opportunity Assessment

The Opportunity Assessment is enough 90% of the time.

  • What business objective are you focused on?
    typically one of the OKR objectives
  • How will you know if you have succeeded?
    typically one of the OKR key results
  • What problem are you solving for our customer?
    do you really know it’s a problem?
  • Who are you solving that problem for?
    typically a target market or persona from the strategy

Internal Press Release

The Internal Press Release is not intended to go public – it’s for internal use only. The anticipated audience is new product’s customers – actually it’s the team, management and stakeholders. It’s typically 1.5 pages maximum and written in “oprah-speak”, not “geek-speak”. Sometimes, 3-4 pages of FAQ for anticipated questions are added. The structure of the internal press release looks like this: heading, summary, problem, benefits, quote from you, customer quote, closing / call to action

Amazon uses Internal Press Releases for big ideas / efforts (e.g. site redesign or moving into new country).

An alternative to the Internal Press Release is the Happy Customer Letter describing the benefits for a customer written by the customer and the CEO-letter describing benefits for the company.

Marty writes on his blog about the internal Press Release and the Customer Letter on his blog: https://svpg.com/the-customer-letter/

Lean Canvas

The Lean Canvas is ideally used for a new business unit, a business line or a startup.

Lean Canvas by LeanStack (Creative Commons Attribution-Share Alike 3.0 Un-ported License)
Lean Canvas by LeanStack (Creative Commons Attribution-Share Alike 3.0 Un-ported License)

The canvas is described at great detail at LeanStack: https://leanstack.com/leancanvas

An alternative to the Lean Canvas is the Opportunity-Solution-Tree. It is introduced by Teresa Torres at her Mind The Product Talk 2017 in London “Critical Thinking For Product Teams“.

Marty talks about the different application areas of the Lean Canvas vs. the Opportunity Assessment on his blog: https://svpg.com/lean-canvas-vs-opportunity-assessment/.

2) PLANNING

User Story Mapping

User Story Mapping helps to visualize and deconstruct the problem or solution space. It provides an holistic view and gives context. Through the collaborative process it encourages shared understanding, identifies holes in thinking and improves planning and estimates. Furthermore it heavily influences prototypes and actually helps to scope releases for the product.

More information can be found in the book „User Story Mapping“ by Jeff Patton: https://www.amazon.de/User-Story-Mapping-Discover-Product/dp/1491904909 or here “Design Thinking in a nutshell – what is it and what’s in for us?

A very good example on how workiva used User Story Mapping can be found here “Interaction Design for Enterprise Teams” by Jason Moore on Slideshare.

Customer Discovery Programs

The basic idea of the Customer Discovery Program is to discover and develop a set of reference customers in parallel with discovering and delivering the product. At the stage where the reference customer signs up for the program there is no product ready to be delivered. The Customer Discovery activity makes only sense for real big efforts, absolutely not for features. Serious enterprise customers are very likely to sign-up because they’re burned by the practices of Oracle, SAP and the likes – sell and run.

The reference customer bought the product without any side deals, runs the product in production and loves it enough to tell the world about it. In Customer Discovery, we’re looking for “Earlyvangelists”. An Earlyvangelist is best characterized by these criterion:

  • They have a problem.
  • They understand they have a problem.
  • They are actively searching for a solution.
  • They have a budget allocated.

See also the definition of the Earlyvangelist at https://steveblank.com/2010/03/04/perfection-by-subtraction-the-minimum-feature-set/.

With the Customer Discovery Program it’s simple to tell if product / market-fit is reached: achieved if 6 reference customer for a single market segment (e.g. industry, geography, …) are found. The product / market-fit product is the smallest possible product that meets the needs of this group. If you find only 4 customers or less this means the product market fit is invalidated and a pivot is needed. Work with 5-6 companies – and not more than 8. Talk to as many as possible – e.g. 50. Select only the most attractive for the Discovery Program, the other customers go into the beta program. Agree with the selected companies to be a discovery partner and ensure the right level of access to people and input. Agree with them to become a public reference if they like the delivered product.

In Enterprise business the goal is to find a single product solution that fits all discovery partners. Again, it’s important that all partners are from one single market segment. In Consumer services you should identify 8-20 Earlyvangelists and agree with them on regular phone calls to synchronize.

Examples for customer discovery: OpenTable (SMB), Symantec (Enterprise), Bazaarvoie (B2B2C), xoopit (Platform Services), Apple (internal tools), BarkBox (Consumer Service)

Marty’s blog on reference customers: https://svpg.com/the-power-of-reference-customers/

3. Ideation

Customer Interviews

Customer interviews are needed to understand your customer, to get rid of assumptions and start working with facts. Marty summarizes the value of interactions with customers in his post “Don’t talk to customers?“. Below are the key questions to answer:

  • Are your customers who you think they are?
  • Do they really have the problems you think?
  • How does the customer solve this problem now?
  • What would it take for them to switch?

Additional Ideation Techniques:

  • Concierge Test (see: https://pdmethods.com/concierge-testing/)
  • Public API’s (let others innovate on your product) – be aware of bad usage: Cambridge Analytica + Facebook
  • Hack Days (directed and undirected)
  • Data Spelunking (Hackathon on data)

4. Prototyping

The prototype should minimize the time by factor 10 to provide something to look at. See more on prototypes at Marty’s blog “Flavors of Prototypes“.

User Prototypes

User prototypes are created very fast and lightweight by nature. The prototypes are used for value testing with a consumer or customer to quickly gather feedback on both, usability and value. Low fidelity user prototypes are used for team internal iterations. Use high fidelity prototypes to show-case internally to executive people. The prototype is usually created by the Product Designer with support from the Product Manager. Ideally, when finished, the prototype could be used as a specification for the Delivery process – “prototype-as-spec”.

Paper prototypes are too limited by nature, use wireframing tools (e.g. balsamiq, axureRP, proto.io, FLINTO, UXPin, marvel, invision, Adobe XD) instead. They allow interactions with the prototype and are no more effort to build.

Feasibility Prototypes

The feasibility prototype validates the solution approach. Usually, the prototype is build by engineering to gain further insights on the implementation and test technical (e.g. scalability, performance) risks. The prototype might not be more than a code fragment or a successful validation of a 3rd party software or API integration. It may also happen that product people are not involved in the prototype at all.

Live Data Prototypes

The purpose of the live data prototype is to collect further evidence pro or contra a product decision. This prototype is more expensive to build than the user prototype, but still far less than the actual product. The prototype is not the real product, it’s usually 5-10% of the real product. It includes quantitative a/b-testing but also qualitative testing and is based on real data. A small amount of real traffic could land on the prototype to collect data. Engineering is typically needed to create the live data prototype within 2 days up to 2 weeks.

A lot of people get excited when they see the live data prototype and tend to confuse the prototype with the real product. But there’s still a significant difference between production ready software and the live data prototype. The real product needs:

  • All required use cases
  • Instrumentation / analytics
  • Test automation
  • Scale and performance
  • SEO work
  • Maintainability
  • Internationalization / localization

A good example of a live data prototype is Amazon’s “Frequently bought together” feature. The idea of building the features was rejected by SVP Marketing. So, strong evidence was needed because it was simply too expensive and risky to productize the feature without further evidence. So, the team decided to build a live data prototype with a small amount of real traffic in a specific product category. The prototype was a/b-tested and the collected data provided a significant uplift in business KPI’s. This is a great example of a high-integrity business case.

Hybrid Prototypes

Hybrid prototypes mix those elements needed to tackle the specific risks at hand. They blend various techniques and are mainly limited by your own creativity.

A good example of an hybrid prototype is Zappos. Zappos solved the problem of female shoppers to buy fashion shoes online. They defined and understood their personas and their key problems with shopping online: 1) returning goods 2) no timely delivery 3) not knowing the size 4) bad product images. Zappos prototyped a potential solution to the persona’s problems by mixing a variety of prototypes: user prototype (appealing front-end), live data prototype (product catalogue and images) and the “Wizard of Oz” (buying the shoes at the shop over the street and delivering to customer). Most important was: the users shouldn’t recognize the prototype character of the solution. Zappos controlled the amount of traffic via AdWords and made sure they could handle the manual part. So, with this mixture of prototypes – that for sure doesn’t scale – they could validate demand, value and usability.

Testing Product Ideas

“Prototype as if you know you’re right, but test as if you know you’re wrong.”

d.school

Marty writes in his blog about “Prototype Testing” more detailed on the various ways to testing.

5. Testing Usability

Usability testing includes interacting with customers, getting their feedback. For the test session, have the prototype ready – up and running and focus on the prepared questions to understand if users have issues using your product. The session may be conducted at your office, the customer’s office or at a mutual convenient location (e.g. Starbucks) or – if not possible – remote via video conference.

Recruiting users in B2B context is done via the customer discovery program, in B2C via AdWords. AdWords allow acquisition of users based on keywords and / or geo-targeting. It’s the most cost-effective solution and easy to stop and restart again. Payment is entirely based on performance.

6. Testing Value

Testing value focuses on three aspects: testing demand (is this really a problem?), testing efficacy (how well does the product solve the problem?) and testing response (how excited are the testers?).

Testing Demand

Testing demand answers the question if people are willing to use the product, if they understand the value and see the product solving a real problem they have. Marty talks more about Desirability Testing on his blog about “Product Validation“. Some techniques for demand testing are described below

Fake Door Test

A fake door test fakes a product feature for the customer. If the customer acts with the fake product a thank-you-message is displayed and sometimes contact information is collected. Furthermore, nothing happens. The goal of the test is to collect data, to measure the click-thru ratio. More information on the fake door test can be found here: http://learningloop.io/plays/fake-door-testing.

Landing Page Test

A landing page test pitches product features, products, product lines or other promises to the customer combined with an explicit call to action. Push traffic on the landing page via e.g. AdWords or other comparable methods. Now, measure the conversion – how many people do actually interact with the landing page and are interested enough to follow the call to action? With the click, nothing happens with the customer other than a friendly thank you message and sometimes the question for contact details. More information on the landing page test can be found here: http://learningloop.io/plays/spoof-landing-pages

Explainer Video

The explainer video shows a high fidelity prototype at work. It’s basically a video of a product demo. It is then distributed and measured like the landing page test above. The goal, again, is to measure demand for the demoed product. More on the explainer video: http://learningloop.io/plays/video-demo

Kickstarter Testing

A great way to test a product idea without jeopardizing your brand is to test demand on kickstarter.com. Just place the product idea at the crowdfunding platform as a “nobody”. If the idea creates enough buzz it’s worthwhile a further investment, if not it can be dropped silently without creating any noise. Read more on the idea from Mark Dwight “How to Kickstart Your Market – Why even established companies can use crowdfunding.

Qualitative Value Testing

“Find everything that’s wrong with the product and fix it; Seek negative feedback.”

Elon Musk

Qualitative testing explains why it’s working or not, it gives insight why something is happening or isn’t. It doesn’t try to prove anything. You won’t get the answer from any one user test; every single test provides another piece of the puzzle. It’s important to test with real users and customers to judge the value.

Qualitative value testing is done with prototypes or the real product. It provides insights from usability and value perspective. On top it usually provides unexpected insights from the customer. It’s typically done fast and cheap. To really understand how much customers value the product, various questions or tests can be conducted:

  • Payment – will they pay for it?
    credit card information, pre-order form, letter of intent (in B2B)
  • Reputation – will they recommend it?
    NPS, introduction to peers or the boss, public reference
  • Time – will they meet again? Will they invest their time?
    agreement on follow-up meeting, non-trivial trial
  • Behavior – will they switch from their current solution?

Quantitative Value Testing

“Features are not inherently valuable. The value for our customers is only realized when a feature fulfills a need. It’s only realized for our business when we see the results of our work move the needle. That’s why we need to concentrate on the outcomes over the outputs.”

Melissa Perri (see: https://melissaperri.com/)

Quantitative value testing can provide evidence or even proof that something truly works – or isn’t. It generally can not explain why it’s so. It’s done to get a clearer picture on the impact on your revenue, your brand, your customers and also your employees.

Quantitative testing can be done with the existing product – or a live-data prototype – in an A/B testing setup on a certain amount of your traffic. Alternatively, it could be done with a limited amount of your customer through invitation. In a B2B scenario, you’d use your existing customer relation via the Customer Discovery Program to get exposure of the test to people.

A good example for a quantitative value test is Spotify’s “Discover Weekly” feature. Data collected in an A/B test was compelling enough to fully implement the feature. The launch-ready implementation included some big hurdles and a lot of effort on the data crunching side. So, it was well worth the effort to test – before – putting the feature in Delivery.

7. Testing Feasibility

Testing feasibility mainly addresses technical concerns – are we able to build this at all? To test feasibility it’s important to create prototypes that focus on the key areas of concern. Emphasize speed of learning over reuse of the written code. Your Tech people need to answer these questions:

  • Do we know how to build this?
  • Do we have the skills on the team?
  • Do we have enough time?
  • Do we have the right architecture or components?
  • Do we understand the dependencies?
  • Will the performance and scale meet our needs?
  • Do we have the infrastructure to test and run this?

8. Testing Business Viability

Business viability – does this feature / product fit with our business? – needs to be addressed to be successful within the own organization. Your stakeholders (e.g. Senior Executives, Sales, Marketing, Finance, Legal, Security, Business Operations, …) need to be informed regularly, you need to earn their trust. Have discussions with them and make them feel you understand them – but remember: everyone has a voice, but not a vote! Try to engage individually with them, group meetings can cause a lot of damage and are harder to handle. When talking to your stakeholder, have your data ready – data always beats opinions. And read the signs during the meeting – differentiate between stop signs and yield signs.

Techniques you can use for a stakeholder meeting is typically a high fidelity user prototype and / or a product walkthrough.

This blog post is part of a series. It summarizes my personal notes of the workshop held by Marty Cagan “How to Create Tech Products Customers Love” from 5th to 6th of June in 2019 in San Francisco.

Product Culture & Transformation – notes on “How to Create Tech Products Customers Love” – #11/11

Product Culture & Transformation - notes on "How to Create Tech Products Customers Love"

Product Culture & Transformation

Transformation Techniques

One transformation technique Marty recommends is the Discovery Sprint. He recommends to do a Discovery Sprint when a team struggles to learn how to do product discovery, when the team has something big and critically important to solve or if the team is just moving too slowly. Marty talks more about it on his blog https://svpg.com/discovery-sprints/ and refers to the Book “Sprint” by Jake Knapp et al.

Another is named Pilot Teams. The idea behind the Pilot Teams is to create success within a smaller protected environment and convince doubtful or fearful or lazy people to follow the change process. The principle is borrowed from the technology adoption curve (aka “Gartner Hype Cycle”) – some people are early adopters, others are less eager. Chris Jones from SVPG talks about this technique in “Pilot Teams“. With these pilot teams the idea of A/B testing – well known from product development – can be applied to organization development as well.

Outcome-based Roadmaps is yet another way to start the transformation process. Simply continue working with product roadmaps, however introduce two differences. First, annotate every roadmap item with its associated expected business result. Every time this item is discussed highlight the expected business result. Second, after the launch of an roadmap item report immediately the actual result vs. the expected result. So, during the next 3-12 month the opportunity assessment information will get its way into the roadmap. For prioritization try to move away from prioritizing ideas to problems.

Common Product Discovery Pitfalls

Marty mentions several pitfalls he experienced and saw teams struggle with. He talks a lot more in his blog post on “Product Discovery: Pitfalls and Anti-Patterns“. Here’s just a summary and some notes.

  • Confirmation-biased Discovery
    The team and / or the stakeholders are not really interested in the results of Discovery, they just need affirmation.
  • Product as Prototype Discovery
    The team pretends working on a prototype implementation but it takes too long to actually get the prototype shipped (e.g. 4 month).
  • Partial Team Discovery
    Not Technology, UX and Product go see the customer, it’s only Product + UX.
  • One-Dimensional Discovery
    The team focusses only on quantitative or qualitative validation and draws wrong or incomplete conclusions.
  • Big Bang Discovery
    The team works on a single, big release shipped within a lengthy time frame. They don’t work in an iterative mode.
  • Outsourced Discovery
    The organization / stakeholders hired a “creative” agency to do the creative Discovery work. The implementation should then be picked up by the team.

Culture Baseline of successful companies

“If we get the culture right, most of the other stuff will happen naturally on its own.”

Tony Hiseh, CEO Zappos
  1. Tackle Risks up Front
    • Value Risk – will they use / buy it?
    • Usability Risk – can they us it?
    • Feasibility Risk – can we build it?
    • Business Viability Risk – will our stakeholders support it?
  2. Define Products Collaboratively, not Sequentially
    • Product Management
    • Product Design
    • Engineering
  3. Focus on Business Results, not Output
    • Product teams exists to solve problems in ways that your customers love, yet work for your business.

This blog post is part of a series. It summarizes my personal notes of the workshop held by Marty Cagan “How to Create Tech Products Customers Love” from 5th to 6th of June in 2019 in San Francisco.

Variable salaries do not motivate – at all

Variable compensation models and motivation – an experience report

We’re quite a young and digital organization. Our shareholder – a big publishing house – demands variable portions of the salary as a motivational factor. If that’s set in stone, you better look how to implement it best. Since 2015, we implemented variants of a variable compensation model and I’d like to share some of our learnings on the various models.

Salaries in the knowledge workers’ world

Today’s salaries usually have at least two components: the fixed part (paid usually every month) and the variable part (paid usually quarterly, twice or once per year). The fixed part represents the compensation for the working hours and fulfillment of the work contract. The variable portion can vary according to the agreed targets. Depending on the degree of achieving the targets the multiplier for the variable portion may vary from 0 to 1.5 or even higher.

Motivation and knowledge worker

My absolute favorite to understand motivation – and the impact of money on knowledge worker is the video by RSA describing Dan Pink’s thinking behind his book “Drive”: https://www.youtube.com/watch?v=u6XAPnuFjJc (10m 47s definitely worth watching!)

Motivation = three major ingredients: Mastery, Autonomy, Purpose.

Mastery is an intrinsic driver of knowledge worker. They want to improve their skills, they want to get better and better. A great workplace environment takes this into account and leaves room for people to practice, practice, practice. Autonomy gives people space to solve the problems they’re working on. Nobody tells them how to do so, nobody’s looking at the output. Only the outcome counts. Purpose sets the work activities into a greater context. Everything has a meaning, people understand why they do what they do. They see the greater picture, the vision. Purpose, in my opinion, is the most important ingredient for motivation. Put all three together as a fundament for a work environment and you have very likely intrinsically motivated people.

Money, on the other hand, as a benefit for work kills motivation. It is nice to receive your variable salary once, twice or 4 times a year. It’s a nice way for your boss to say “Well done, thank you!”. BUT the money is not a long lasting instrument to create or increase motivation (look at 2:12 into the video – they present research results there …).

So, why do we still have variable salaries to push motivation and output?

I know quite some young companies, some startups, some larger organizations who operate 100% on fixed salaries. They understood the basic principle of motivation and compensation. On the other side there are still some old, traditional and rusty organizations with variable compensation plans. Some decades ago, they connected the compensation to personal goals and never questioned themselves. Or it’s so common, they can’t even think about getting away from this model.

2015 – The “yearly revenue and individual targets”-model

Management sets a revenue target and every employee and his/her manager agrees individually on targets. The thesis behind the model: money is a key motivator to achieve top performance. The variable part of the salary is 100% connected to a company revenue goal and one or many individual goals.

We observed lengthy and excessive negotiations on individual goals with our employees. Even worse, the just-agreed goals hold just 4-6 weeks until they need refinement. Furthermore, we observed individuals stating they couldn’t help each other because this action would directly conflict with them achieving their goals. In essence, the model leads to people optimizing their personal benefits and defocusses company goals.

2016 – The “yearly revenue and department targets”-model

Management sets a revenue goal and each department head sets a yearly department target. The department target (e.g. “Reduce page load time to less than 2 seconds”) holds for the whole year and is always present. It will influence the way people work together but is not always a focus topic. The department target occurs in the variable compensation plans for all department employees. Thesis behind this model is again: money is a key motivator to achieve top performance. The variable portion of the salary is again connected 100% to the company revenue goal and one or many department goals.

Applying this model we observed less conflicts between individual employees. It’s quite time consuming during the identification of the department goals. They need strong alignment amongst each other. We managed to achieve quite good alignment – however had some occasions where people ended in conflicting department goal discussions. In the end the whole staff focussed ways more on achieving the department goals – better than with the previous model. However, the organization didn’t “feel” aligned on joint goals, more trying to achieve the department goals. Producing shiny winners on the cost of the overall company targets.

2017 – The “yearly revenue and quarterly company targets”-model

Don’t name the model OKR. We successfully burned OKR in 2014. Tried to implement it without external, experienced help. It ended in a process-by-the-book implementation and a perception of a grass-root democratic, inefficient and cluttered way to set 4 management and 4 team targets per quarter.

In this model, the management sets a yearly revenue target and company targets for the next 3 month. The company targets include everybody in the company – no matter what function or department. The targets need to be S.M.A.R.T. (Specific, Measurable, Achievable, Relevant, Timebound) and led to quite some discussion during definition – both between department heads and employees. The targets include e.g. specific product features, specific sales products or marketing activities. Thinking behind this model: everybody sits in the same boat and again money is a key motivator to achieve top performance. The variable portion of the salary is connected 100% to the company revenue goal and quarterly company goals.

During the implementation of this model we found “same company, same targets” led to some motivation for the active influencer for the goals (e.g. product, technology, sales, marketing) and quite some frustration for those having no influence at all (e.g. finance, HR, administration). They had to rely on their colleagues delivering the best job possible. On the other hand, we were able to set relevant goals for the next 3 month and were able to steer the company in a clever and agile way through the rough sea of economical changes. We had the staff members being focussed on achieving few topics of high relevance for the company.

We also learned – the hard way – that goal achievement communication can lead to some confusion and irritation if not done 100% transparent. And setting quarterly goals leads to quite some overhead to define goals every 12 weeks. Agility comes with a price tag!

2018 – The “100% guaranteed and 150% possible”-model

2017 ended with some really bad environmental messages for our business model. We needed to change quite some things. Amongst them was the variable compensation model. Our ambition at the time was to bring maximum calm to the staff members and allow them to focus on the company’s focus goals. One measure was to put away the variable portion of the compensation model. We guaranteed 100% of the bonus and made 150% possible if we achieved a specific traffic target earlier than we expected it.

Thinking behind this model is (see Dank Pink above): money doesn’t have any influence on work performance, but it has on work morale. We decoupled the compensation from achieving targets – allowing people to work on the company’s focus topics. And as a bonus, there is this 150% stretch goal. Nice to achieve – and desirable – but it just sits there.

We observed almost no discussion on compensation and fairness / unfairness of goals. People focused on getting the job done. I’d refer to the state of people as “intrinsic motivated”. At year end we didn’t manage to catch the 150% goal which led again to some frustration amongst the team. Furthermore, some specific departments (e.g. sales) perceive “100% fixed” less as an adorable state. For them it’s less motivating since their working model always follows includes “catching numbers”.

2019 – The “revenue and EBIT goal”-model

Beginning 2019 we found ourselves in a more stable environment and switched back to a variable compensation model. This time, we decided to focus on setting company wide financial goals. Everybody is able to influence them and the effort setting them is limited. The goal is set end 2018 for the whole year.

The thinking behind this model: money doesn’t have any influence on work performance, but it has on work morale. At a first glance, the model doesn’t look like an advancement but it effectively decouples financial goals from specialist topics. It’s the same for everybody and done once a year. So far, so good. It turns out to be a bit problematic since the environmental conditions changed quite a bit and the target corridors defined end 2018 are no longer achievable. So an adjustment is needed!

Which model worked best so far?

The 2015, 2016 and 2017 models worked better and better each year. Still having significant flaws with the direct connection between motivation and money paid. But the got better.

2018 was the most successful model so far. Less friction, lots of motivation and high pace – outcome over output. But we also had some frustration in performance-oriented departments (e.g. sales).

2019 doesn’t feel like an advancement from 2018. But 2019 is not over – let’s see.

The holy grail? Well, I don’t think we found it – so, we need to move on and adapt.

Image credentials: Thx! https://pxhere.com/en/photo/1453161

When to use waterfall, when agile?

Software projects failed a lot in the past. They failed to deliver the value for the business, were too late or ways out of budget. The selected process method was usually the scapegoat for the failure with agile methods being the answer to any question in software development. But as usually in live, it’s not black or white. The selection of the right software process method depends on the surrounding of the project. I gathered some industry input and combined it to reflect the current thinking regarding agile software development methods vs. traditional methods.

The adapted Stacey matrix

Adapted Stacey Matrix for technology / software development environment

The original stacey matrix supports decision making processes suggesting appropriate management actions and defines four areas: simple, complicated, complex and chaotic. The suggested actions depend heavily on the context of the decision making.

The dimensions of the adapted matrix: HOW and WHAT

The x-axis of the adapted matrix deals with the HOW. If the team knows the technology well and has used it many times before, we’re on the left. Otherwise, if the technology is completely new to the team we’re on the right of the dimension. The y-axis positions the WHAT. On the bottom of the axis, the stakeholder of the project all agree on the goals and have the same understanding of the expected outcome. On top it’s the opposite, no agreed requirements and no alignment on expectations. The individual mix of the project points to a certain area with a process model suggestion in the adapted matrix.

Waterfall …

Waterfall is a traditional project management method with sequential steps and no iterations. Massive upfront planning is done before any implementation work starts. If all goals and steps are clear, waterfall produces consistent results in a predictable and repeatable way. The clearly defined tasks lead to an optimized sequencing and optimal resource allocation. Waterfall optimizes resources and return on invest if cause and effects are clear to anybody in the project team.

… vs. Agile

Agile stands for SCRUM, Kanban and LEAN methods with flexibility, quick response and constantly changing environments in mind. They start quicker with smaller scope for the current increment with the scope being like a rolling window. Uncertainty of the projects’ goals needs quick adjustment and adaptation during the whole execution. Only close and frequent collaboration with all team members make agile projects successful. If causes and effects aren’t clear, agile works in small steps towards a value-generating and broadly accepted result.

Simple to chaotic – from “known knowns” to “unknowables”

Simple = easily knowable, the known knowns

Projects in the simple zone unveil very few surprises, decisions are fact- or evidence-based, advancement occurs in orderly, sequential steps and the WHAT is clear to anybody. Any size projects with clear activities and repeatable results fits in this category. It has been done multiple times before and best practices exist as benchmarks. The process is simple and could be handled in a check-list style.

Going forward in a simple, fully predictable project means reducing it to the maximum to make the single pieces easier to understand. Examples of simple: recipes, tasks on an assembly line, checklist based work.

Complicated = not simple but still knowable, the known unknowns

The complicated zone segments into socially and politically complicated and technically complicated. Complicated means less simple but still somewhat predictable.

In Social/political complicated environments people can not agree on the purpose of the project and the expectation on results is not clear. Requirements are conflicting amongst the diverse stakeholder which could be resolved with waterfall to get clearance on WHY before WHAT before HOW. On the other hand applying agile techniques could help convincing stakeholders to agree on already achieved results and smoothing the further requirements discussion. The project team needs to pay special attention on getting early agreement between stakeholders in place.

In technically complicated contexts it’s clear on WHY and WHAT to achieve. Still, the HOW is not clear. An agile iterative approach helps getting feedback from the project team on the achievements making adaptations possible.

Going forward in a complicated project means as well reducing it to the maximum to make the single pieces easier to understand. Technically complicated is e.g. using a specific technology for the first time. Political/social complicated is e.g. if the relation between cause and effect are not clear enough or conflicting opinions amongst various stakeholder exist.

Complex = not fully knowable but reasonably predictable, the unknown unknowns

The complexity zone stands for high risk and uncertainty and requires a high feedback frequency. Neither requirements nor the execution are clear. Holistic defined process methods don’t work any longer. The context asks for a more explorative approach with transparency, frequent inspection and adaptation. SCRUM as a process method in the toolbox of the agile mindset is the method of choice. It increases transparency with small iterations and frequent check-points allowing cheap adaptations. The team planning is the start point for each new iteration and allows immediate feedback from stakeholders to the teams to adapt the next iteration.

Complexity can not be reduced, some understanding can be achieved and complexity can not be planned, it simply grows. A good example of a complex project is software development in general. The requirements are rarely fully defined right at the beginning and it’s seldom clear which architectural solutions are superior to others.

Chaotic = neither knowable nor predictable, the unknowables

In chaotic zone requirements and execution path are both undefined and the risk is high. Kanban as the most flexible project management method is the tool of choice. With no structure like sprints and the only focus on work in progress (WIP) Kanban focuses on continuous delivering results to allow further modifications in direction and backlog items.

The goal is to move from chaotic towards complex by dividing the problems. The principle “Act, Sense and Respond” helps navigate towards the zone of complexity.

Sources – from where I learned:

Innovation and organizations – Hackathon vs. RocketLab

Innovation is usually part of agile product development methods. Sometimes, however, agile methods just replace other methods. SCRUM replaces Waterfall, KANBAN formalizes previously unordered work. Obviously, the innovation dilemma remains still open. Where comes the creativity from? The ideas? Where to test those hypotheses which are not part of the daily routine?

The hackathon as innovation tool

Some organizations run hackathons once or multiple times a year. We did and are doing this as well. We organized already 6 hackathons in the past. Once yearly. Did we see the innovation boost? Well, yes – and no.

How do we organize a hackathon?

A hackathon at gutefrage.net is a timeboxed activity (usually 2,5 days) and surrounded by a lot of social activities. We cook, we bake, we experience Virtual Reality, we do some board games, we play the football table and have a good time and fun. The whole company participates usually and is excited to validate hypotheses which are usually not part of the product development. There are no limits from a topic perspective. Teams organize themselves via a democratic voting exercise right at the beginning. People pitch their ideas and convince other people to become part of this specific project group.

What’s the typical outcome?

During the hackathons at gutefrage.net one out of five ideas launch during the hackathon. This one idea is production ready and creates value right from the launch. The other ideas typically proof aspects, create prototypes of various qualities, cover maximum the best-case implementation and still need an investment of 80% to be ready. The hackathon is a great team building event, it’s great for the morale, the culture. The hackathon drives people’s motivation and frustrates them if their project doesn’t make it into the finals.

What issues do we see?

The hackathon validates some hypotheses, some not and the question remains open what to do with all the started work? Will we follow some traces? Will we just abandon the work? Needless to say – after the 2,5 days hackathon there waits daily business in form of agile software development work. At gutefrage.net we promised to launch the winning idea and typically abandon the remaining work. We found that’s not the most efficient way to drive innovation.


The RocketLab – our way to innovate

As a learning organization we drew some conclusions from the hackathon experience. Mixed teams with participants from all relevant areas worked very well. The one project going live was a real push for team and company motivation. The others weren’t as good for morale. Thinking about this for a while we came up with a slightly different format – the RocketLab.

What’s different between RocketLab and the hackathon?

The RocketLab stands for outcomes and can potentially solve any topic: Product, Technology or others of cross-discipline relevance. At day one of the RocketLab there is one specific hypothesis the team focuses on. The team exclusively works for a defined period solely on solving this one issue. No distraction, just 100% focus. The team contains all disciplines to solve the issue on hand. They are all committed to create the best solution possible within the given time budget. It’s a team of maker, not talker, not theorists, no visionaries. The hackathon is broad and unspecific by nature, the RocketLab has a given goal to accomplish with the solution delegated 100% to the team.

How do we organize a RocketLab?

It’s typically either Product or Technology bringing up a specific hypothesis (or a technical complex problem to solve). A short discussion determines the amount of time we’re willing to spend on finding a solution – usually 3 to 5 days. The organizers invite people to participate in the RocketLab and the Lab kicks off. It’s never the whole company, only few people but interdisciplinary.

The initial task after kick-off is an intense planning session. The organizers introduce the hypothesis to a greater detail and the team sets goals – together with metrics. Right afterwards with a clear goal in mind and a good understanding of the metrics solution ideation starts. Ideally, the team ends this activity with a solid set of tasks for each team-member.

The RocketLab needs 100% dedicated team members – no excuses – and sits co-located in a special meeting room.

What’s the typical outcome?

The expected outcome of the RocketLab is a solution for the hypothesis from the beginning. The solution is live, up and running. If the team was not able to solve it 100% they have a clear understanding of the remaining efforts and a thorough plan. The plan is then executed in regular agile development work. One hypothesis, one implementation, one proof. The RocketLab is an efficient and effective tool to work concentratedly on a hypothesis – goal-oriented but very intense.

What issues do we see?

We did around 10 RocketLabs for very different topics. Very concrete, technical topics up to very abstract conceptual work. The results were sometimes simply spot on, other times needed further perfection during daily work. In essence, the RocketLab is a tool which borrows aspects from the hackathon but is more effective and efficient. It simply works for us and produced some very surprising solutions.

We still see some issues with the spill-over effect of the RocketLab – but that’s a minor problem. Only 20% of the Labs experienced the spill over.

graphics
rocket – used under creative commons license (CC BY 4.0), non-modified
hackathon logo “The Hacking Dead”: © by gutefrage.net

Toolbox, Best Practice, Tools, HowTo’s – agile practices and how to apply them

The internet holds quite a lot of information. Here’s my favorite collection of tools, toolboxes, methods, best practices and howto’s from various fields of application. Most of them have a tight coupling to agile software development, agile organization development, product development and cross those fields.

Following a list of links and a short description of what to find on the site.

Open Practice Library, https://openpracticelibrary.com/

The site contains methods and tools around product Discovery and Delivery practices. The methods are collected by the community and serve to inspire seeking minds to test them in various situations. The methods are positioned in 4 main areas: Discovery, Delivery, Options Pivot and the Foundation.

Discovery contains methods like:

Options Pivot holds tools like:

Delivery contains these tools:

Foundation hold something like these:

Ideo – Design Kit, http://www.designkit.org/methods

This is a collection of tools around Inspiration, Ideation and Implementation. The kit is provided by IDEO.org.

Inspiration

Ideation

Implementation

Other Tool Sets