Measuring your product performance – PULSE vs. HEART

PULSE vs. HEART – an overview

Tracking of product metrics occur on various levels. PULSE and HEART stand for a logical structure of metrics to measure several aspects of your web product performance. PULSE reflects a more low-level and direct approach to performance figures. HEART on the other hand focuses on the customer experience.

Find an in-depth description of both performance frameworks in this paper from Google “Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications” by Kerry Rodden, Hilary Hutchinson and Xin Fu.  How did I came across the HEART framework? Well, I found this inspiring talk by Roan Lavery, co-founder and CPO at freeagent: “Driving Growth vs. Building Core Value” at a mindtheproduct conference in 2018. In the video he describes – amongst other aspects – how they applied the HEART framework at freeagent.

PULSE – low-level and direct

The PULSE framework focuses heavily on direct impact KPI’s to measure the performance of large-scale web products. They typically reflect technical or business aspects of the performance. PULSE stands for Page views, Uptime, Latency, Seven-day active users and Earnings. 

Page views reflects the amount of users visiting your site. Uptime gives the percentage of time the server infrastructure is up, running and serving content. Latency gives a proper indication of the performance of your site infrastructure and your overall software development efforts on execution speed. Seven-day active users says a lot about retention – the ability of your site or product to motivate people to come back multiple times within 7 days. Even if seven-day active users looks like an user centric KPI it doesn’t tell anything about the level of satisfaction of your users. Earnings, finally, gives a good indication if the product works – or not.

These dimensions of a product are definitely worthwhile watching and should be observed thoroughly. But – are they good candidates to focus on user centered product development? Are they any good when it comes to value generation?  

HEART – higher level and user focused

The HEART framework is less generic, it costs more work to identify the right metrics – but it helps a lot to focus on users and makes value generation the most important goal. HEART is more adjusted to the individual product, it’s less direct and needs a good understanding of the product to measure. HEART stands for Happiness, Engagement, Adoption, Retention and Task Succes. 

Happiness, Engagement, Adoption, Retention and Task Success in a nutshell

Happiness is a very fluffy description of a very important state of mind of product users. If the product touches people, if it really helps it makes user more happy. A variety of KPI’s express the happiness of your users. The KPI’s are very product specific – in our case we took “Net Promoter Score”, “User Survey”, “#Bugs on the board” and “Upvote index” to measure this very qualitative dimension.

Engagement measures the level of engagement of users with your site. It’s not overall engagement with a site in general, it’s engagement with the core aspects of the site. Focus is on specific pages and sections that are critical for the value perception of the user. We track “PI per Visit”, “Engagement on QDP” (our most important page type), “User activity”.  

Adoption focuses on the amount of new users discovering the product and actually decide to become active. We decided to go with “#Registrations” and “Daily Activation”.

Retention measures the amount of users coming back to the product and use the product over a period of time. We measure retention with “Stickiness 30d” and “Churn Rate”.

Task Success measures the amount of tasks completed by the users. Not any tasks – those tasks providing most value to our users. It’s important to understand how many users are really engaged with the product and perceive value from the most valuable functions of the site. We are looking at “Q&A Index” (ratio of answers and questions per user), “Time 2 Answer” and “HA Ratio” (ratio of helpful answers to all answers).

Final Thoughts

Engagement, Adoption and Retention metrics are typically measured over specific periods of time. For some products it might be worthwhile to focus on a 7-day-period others might need a 30-day-period.

We’re still fresh on HEART but I strongly believe it will change the way we develop our product in future. 

Where to start to speed up your IT environment – Here are 5 areas to look after.

Anil Cheriyan shared his thoughts on where to focus to create a fast and better working IT environment in the financial services industry – to speed up the organization (see: https://enterprisersproject.com/article/2017/6/suntrust-cios-formula-speed-relies-cloud-devops). In his post he mentioned 5 areas you should look after to break with old habits and start creating a fast pacing environment.

Anil Cheriyan is Director/Deputy Commissioner, Technology Transformation Services for the U.S. Federal Government. Previously, he was managing partner of Phase IV Ventures, a consulting and advisory firm.

Cloud

Two important aspects associated with the terminology “Cloud”. First it’s important to understand the implications of the various cloud strategies (ranging from private cloud over hybrid constructs towards public clouds). Get your strategy clear on which areas to host where. Criteria to look at are: business value provided, business continuity, resilience, security. Second aspect is the organization. Get your people involved. They need to participate in the strategy definition. They will execute them actually. No time for information hiding and bimodal IT infrastructures.

Modular architecture

Getting towards a modular architecture introduces flexibility in decisions, eliminates bottle necks and allows a decentralized governance. Today’s architectures are still monoliths or more advanced SOA stacks or somewhere in between. A more modular architecture exposes API’s via micro services. This architecture allows distributed ownership models. Complex is actually the implementation of these architecture rewrites. A lot of business related activities and the re-architecture work is a hard effort to combine.

DevOps

DevOps is finally all about the mindset of people and the break-up of silo-ed organizations. People need to learn and understand the importance of collaboration and trust. This sounds simple, turns out to be a heavy change project. Anil started pilot projects and introduced the true DevOps mindset and collaboration through success cases. It’s not about adopting rules and processes from the DevOps movement “by the books” – it’s about training your talent to work closer together.

Agile development

Agile development in software development is quite wide spread and commonly used. The acceptance over waterfall models is – where appropriate – high. Issues occur if the agile software development processes get surrounded by traditional waterfall-oriented functions – control functions. The most challenging part is to get agility into release management, deployment and integration testing.

Design thinking

Most important aspect of design thinking is the customer centricity. Understanding the real problems of the user to be solved is at the core of the approach. Not hunting the 100% perfect solution with all nice and “useful” features. Going for the most valuable solution, ship it fast. This requires heavy re-thinking within the organization. It’s more about talent and collaboration models. Important is to get people together with a thorough understanding of the industry and processes to help solving the customer’s pain points.

Are you a mercenary or a missionary leader?

Ever asked yourself what you’re after eventually? Are you working being “involved” or “committed”? Are you a mercenary or a missionary? What’s your leadership style and where does it make a difference. All these questions were picked up by John Doerr (Kleiner Perkins Caufield & Byers) in April 2000 and put in relation.

A great article to read: http://knowledge.wharton.upenn.edu/article/mercenaries-vs-missionaries-john-doerr-sees-two-kinds-of-internet-entrepreneurs/

In Doerr’s view there’s a fundamental difference between mediocre organizations and real value driving – great – ventures. He mentioned five dimensions to look at:

  1. led by missionaries, not mercenaries
  2. top-notch, passionate leadership
  3. operation in large, rapidly growing and under-served markets
  4. reasonable financed
  5. working with sense of urgency

Especially the missionaries vs. mercenaries caught my attention. Marty Cagan from Silicon Valley Product Group wrote about the importance of people’s attitude in product development with a reference to missionaries vs. mercenaries (https://svpg.com/missionaries-vs-mercenaries/).

How does Doerr characterize the two M’s?

mercenarymissionary
driven byparanoiapassion
thinkingopportunisticallystrategically
going forthe sprintthe marathon
focus ontheir competitors and
financial statements
their customers and
value statements
arebosses of wolf packsmentors or coaches of teams
worryabout entitlementsabout making a contribution
motivationlust for making moneydesire to make a meaning

Where are you? And more important – where do you want to be?

Understanding the difference is equally important for leaders in organizations as for leaders in technology or product development.

Architecture alternatives for rendering a web site

There’s a great overview of technologies available from Google comparing the different architecture options to render a web site. Jason Miller and Addy Osmani present options from SSR (server-side rendering) over various mixed models to complete CSR (client-side rendering). They describe the pros and cons of the various approaches and give hints on what to use in which situation. A great read!

https://developers.google.com/web/updates/2019/02/rendering-on-the-web

Rendering options

  • SSR: Server-Side Rendering – production of HTML is done on the server
  • CSR: Client-Side Rendering – creation of HTML is done on the client, usually using the DOM
  • Rehydration: Using a JavaScript based client app to show the server-rendered HTML – mixed with the DOM tree and associated data
  • Prerendering: generation of HTML is done during build time

Performance acronyms

  • TTFB: Time to First Byte – time between clicking a link and the first bit of content
  • FP: First Paint – time until any pixel gets visible to the user
  • FCP: First Contentful Paint – time until the requested content (article, body, ..) becomes visible
  • TTI: Time To Interactive – time until a page becomes interactive

Jason and Addy wrap their great article up with an overview of the options. Since it’s presented under Creative Commons Attriubtion 3.0 License I decided to reproduce it here for further reference.

Toolbox, Best Practice, Tools, HowTo’s – agile practices and how to apply them

The internet holds quite a lot of information. Here’s my favorite collection of tools, toolboxes, methods, best practices and howto’s from various fields of application. Most of them have a tight coupling to agile software development, agile organization development, product development and cross those fields.

Following a list of links and a short description of what to find on the site.

Open Practice Library, https://openpracticelibrary.com/

The site contains methods and tools around product Discovery and Delivery practices. The methods are collected by the community and serve to inspire seeking minds to test them in various situations. The methods are positioned in 4 main areas: Discovery, Delivery, Options Pivot and the Foundation.

Discovery contains methods like:

Options Pivot holds tools like:

Delivery contains these tools:

Foundation hold something like these:

Ideo – Design Kit, http://www.designkit.org/methods

This is a collection of tools around Inspiration, Ideation and Implementation. The kit is provided by IDEO.org.

Inspiration

Ideation

Implementation

Other Tool Sets

Do’s and Don’ts working with OKR’s

OKR stands for Objectives Key Results and is a tool set to streamline an organization towards the most important goals for a given time period (e.g. a quarter). Google Venture’s OKR-expert Ken Norton talks about some of his observations on organizations implementing OKR’s in this interview “Are you doing OKR’s right?”.

Ken Norton gained his expertise working with 300+ companies from the GV’s portfolio. OKR’s are one great tool to align an organization to work toward the most important goal at the time. Here’s a bullet list of my key learnings:

  • Communicate OKR’s clearly
  • OKR’s are the CEO’s tool to set the direction for the organization
  • Very, very few in numbers
  • OKR’s are measurable – you know when you’re there
  • OKR’s keep the organization ambitious
  • An organization of less than 100 people doesn’t need more than the organization’s OKR’s
  • OKR’s do only work if the culture of the company allows for them
  • They’re most effective if a combination of top-down and bottom-up
  • OKR setting surfaces organizational problems – deal with them