measure

Measuring Quality

At the end of last year, Cassie raised the question of ‘how to measure quality?’ on our metrics mailing list, which is an excellent question. And like the best questions, I come back to it often. So, I figured it needed a blog post.

There are a bunch of tactical opportunities to measure quality in various processes, like the QA data you might extract from a production line for example. And while those details interest me, this thought process always bubbles up to the aggregate concept: what’s a consistent measure of quality across any product or service?

I have a short answer, but while you’re here I’ll walk you through how I get there. Including some examples of things I think are of high quality.

One of the reasons this question is interesting, is that it’s quite common to divide up data into quantitative and qualitative buckets. Often splitting the crisp metrics we use as our KPIs from the things we think indicate real quality. But, if you care about quality, and you operate at ‘scale’, you need a quantitative measure of quality.

On that note, in a small business or on a small project, the quality feedback loop is often direct to the people making design decisions that affect quality. You can look at the customers in your bakery and get a feel for the quality of your business and products. This is why small initiatives are sometimes immensely high in quality but then deteriorate as they attempt to replicate and scale what they do.

What I’m thinking about here is how to measure quality at scale.

Some things of quality, IMHO:

axeThis axe is wonderful. As my office is also my workshop, this axe is usually near to hand. It will soon be hung on the wall. Not because I am preparing for the zombie apocalypse, but because it is both useful as a tool, and as a visual reminder about what it means to build quality products. If this ramble of mine isn’t enough of a distraction, watch Why Values are Important to understand how this axe relates to measures of quality especially in product design.

toasterThis toaster is also wonderful. We’ve had this toaster more than 10 years now, and it works perfectly. If it were to break, I can get the parts locally and service it myself (it’s deliberately built to last and be repaired). It was an expensive initial purchase, but works out cheap in the long run. If it broke today, I would fix it. If I couldn’t fix it for some extreme reason, I would buy the same toaster in a blink. It is a high quality product.

coffeeThis is the espresso coffee I drink every day. Not the tin, it’s another brand that comes in a bag. It has been consistently good for a couple of years until the last two weeks when the grind has been finer than usual and it keeps blocking the machine. It was a high-quality product in my mind, until recently. I’ll let another batch pass through the supermarket shelves and try it again. Otherwise I’ll switch.

spatulaThis spatula looks like a novelty product and typically I don’t think very much of novelty products in place of useful tools, but it’s actually a high quality product. It was a gift, and we use it a lot and it just works really well. If it went missing today, I’d want to get another one the same. Saying that, it’s surprisingly expensive for a spatula. I’ve only just looked at the price, as a result of writing this. I think I’d pay that price though.

All of those examples are relatively expensive products within their respective categories, but price is not the measure of quality, even if price sometimes correlates with quality. I’ll get on to this.

How about things of quality that are not expensive in this way?

What is quality music, or art, or literature to you? Is it something new you enjoy today? Or something you enjoyed several years ago? I personally think it’s the combination of those two things. And I posit that you can’t know the real quality of something until enough time has passed. Though ‘enough time’ varies by product.

Ten years ago, I thought all the music I listened to was of high quality. Re-listening today, I think some of it was high-quality. As an exercise, listen to some music you haven’t for a while, and think about which tracks you enjoy for the nostalgia and which you enjoy for the music itself.

In the past, we had to rely on sales as a measure of the popularity of music. But like price, sales doesn’t always relate to quality. Initial popularity indicates potential quality, but not quality in itself (or it indicates manipulation of the audience via effective marketing). Though there are debates around streaming music services and artist payment, we do now have data points about the ongoing value of music beyond the initial parting of listener from cash. I think this can do interesting things for the quality of music overall. And in particular that the future is bleak for album filler tracks when you’re paid per stream.

Another question I enjoy thinking about is why over the centuries, some art has lasting value, and other art doesn’t. But I think I’ve taken enough tangents for now.

So, to join this up.

My view is that quality is reflected by loyalty. And for most products and services, end-user loyalty is something you can measure and optimize for.

Loyalty comes from building things that both last, and continue to be used.

Every other measurable detail about quality adds up to that.

Reducing the defect rate of component X by 10% doesn’t matter unless it impacts on the end-user loyalty.

It’s harder to measure, but this is true even for things which are specifically designed not to last. In particular, “experiences”; a once-in-a-lifetime trip, a festival, a learning experience, etc, etc. If these experiences are of high quality, the memory lasts and you re-live them and re-use them many times over. You tell stories of the experience and you refer your friends. You are loyal to the experience.

Bringing this back to work.

For MoFo colleagues reading this, our organization goals this year already point us towards Quality. We use the industry term ‘Retention’. We have targets for Retention Rates and Ongoing Teaching Activity (i.e. retained teachers). And while the word ‘retention’ sounds a bit cold and business like, it’s really the same thing as measuring ‘loyalty’. I like the word loyalty but people have different views about it (in particular whether it’s earned or expected).

This overarching theme also aligns nicely with the overall Mozilla goal of increasing the ‘number of long term relationships’ we hold with our users.

Language is interesting though. Thinking about a ‘20% user loyalty rate’ 7 days after sign-up focuses my mind slightly differently than a ‘20% retention rate’. ‘Retention’ can sound a bit too much like ‘detention’, which might explain why so many businesses strive for consumer ‘lock-in’ as part of their business model.

Talking to OpenMatt about this recently he put a better MoFo frame on it than loyalty; Retention is a measure of how much people love what we’re doing. When we set goals for increasing retention rate, we are committing to building things people love so much that they keep coming back for more.

In summary:

  • You can measure quality by measuring loyalty
  • I’m happy retention rates are one of our KPIs this year

My next post will look more specifically about the numbers and how retention rates factor into product growth.

And I’ll try not to make it another essay. ๐Ÿ˜‰

8 thoughts on “Measuring Quality”

    1. I struggled with that book a couple of times, and haven’t ever finished it. But maybe I will try again. I don’t think I gave it the space it required.

      1. Struggled? As I remember it I found it quite accessible. But that alone of course don’t make it good.

        And responding to your post: You seem to equate quality with popularity. And I would very much disagree with that. It is possible that you want to measure popularity/loyalty, just don’t fool yourself by calling it quality.
        (In the examples you give from your house, it seems that you equate availability with quality, but it is certainly possible that you left out arguments for why the items are good quality other than the one you gave, which reads a bit like “they are what I currently have and I like it”)
        One reason something might be popular is if you don’t know that something better exist.

        1. Re: struggled – both attempts were soon after my kids were born, and I was reading small chunks at a time and kept loosing the thread. In short, I would read a page then fall asleep. On your nudge I will find it again later today. ๐Ÿ˜‰

          Re: popularity – I don’t think mass appeal is required to build something of quality at all. I love small scale, local, hand-made, artisan things and personally lean towards disliking a lot of things that are popular. Re-reading my text, the only mention of popularity was to say it isn’t a good enough indicator of quality. But if ‘popularity’ feels implied by my talk of scale, that’s not my goal.

          If your topic of interest is ‘impact at scale’ in a given field (like Webmaker’s is for Digital Literacy for example), then mass appeal is important, and the additional measure of loyalty helps you understand if your overall program to improve digital literacy globally is of high quality.

          But to be clear, high quality ‘things’ can be exclusive to a tiny audience or niche interest too. Though sometimes in that area, exclusivity and price can mask things of low quality. Again I think this is where loyalty as a measure helps distinguish between fads and things of lasting value.

          Re: examples – I don’t follow the ‘equate availability with quality’ note, but I suspect I’m not reading my text cold enough. I was trying to keep those examples short and probably missed a lot of detail. I could write for a few more hours on what I like about those items, but the key point is that even with competing products available, I would choose those examples again because they have earned my loyalty.

          Re: something better could exist: I totally agree, and in that case I expect people won’t be loyal for very long to a product that simply lacks good competition. Especially if the cost of switching is low. Earning loyalty is about more than providing good enough utility for now.

  1. I really liked this essay and all of its tangents.

    I think a key part of the retention metric for teach.mozilla.org is going to be having fresh content regularlyโ€”a reason for people to return to the site beyond their initial introduction. Wondering if you think that’s also a part of the definition of quality.

    1. Thanks Hannah! And yes, I think that captures it exactly. If people keep coming back and using the site over time, we know we’ve built something of quality. ๐Ÿ™‚

Leave a Reply

Your email address will not be published. Required fields are marked *