In my last post I spent some time talking about why we care about measuring retention rates, and tried to make the case that retention rate works as a meaningful measure of quality.
In this post I want to look at how a few key metrics for a product, business or service stack up when you combine them. This is an exercise for people who haven’t spent time thinking about these numbers before.
If you’re used to thinking about product metrics, this won’t be new to you.
I built a simple tool to support this exercise. It’s not perfect, but in the spirit of ‘perfect is the enemy of good‘ I’ll share it in it’s current state.
Optimizing for growth isn’t just ‘pouring’ bigger numbers into the top of the ‘funnel‘. You need to get the right mix of results across all of these variables. And if your results for any of these measurable things are too low, your product will have a ‘ceiling’ for how many active users you can have at a single time.
However, if you succeed in optimizing your product or service against all four of these points you can find the kind of growth curve that the start-up world chases after every day. The referrals part in particular is important if you want to turn the ‘funnel’ into a ‘loop’.
Depending on your situation, improving each of these things has varying degrees of difficulty. But importantly they can all be measured, and as you make changes to the thing you are building you can see how your changes impact on each of these metrics. These are things you can optimize for.
But while you can optimize for these things, that doesn’t make it easy.
It still comes down to building things of real value and quality, and helping the right people find those things. And while there are tactics to tweak performance rates against each of these goals, the tactics alone won’t matter without the product being good too.
As an example, Dropbox increased their referral rate by rewarding users with extra storage space for referring their friends. But that tactic only works if people like Dropbox enough to (a) want extra storage space and (b) feel happy recommending the product to their friends.
- Build things of quality
- Optimize them against these measurable goals
At the end of last year, Cassie raised the question of ‘how to measure quality?’ on our metrics mailing list, which is an excellent question. And like the best questions, I come back to it often. So, I figured it needed a blog post.
There are a bunch of tactical opportunities to measure quality in various processes, like the QA data you might extract from a production line for example. And while those details interest me, this thought process always bubbles up to the aggregate concept: what’s a consistent measure of quality across any product or service?
I have a short answer, but while you’re here I’ll walk you through how I get there. Including some examples of things I think are of high quality.
One of the reasons this question is interesting, is that it’s quite common to divide up data into quantitative and qualitative buckets. Often splitting the crisp metrics we use as our KPIs from the things we think indicate real quality. But, if you care about quality, and you operate at ‘scale’, you need a quantitative measure of quality.
On that note, in a small business or on a small project, the quality feedback loop is often direct to the people making design decisions that affect quality. You can look at the customers in your bakery and get a feel for the quality of your business and products. This is why small initiatives are sometimes immensely high in quality but then deteriorate as they attempt to replicate and scale what they do.
What I’m thinking about here is how to measure quality at scale.
Some things of quality, IMHO:
This axe is wonderful. As my office is also my workshop, this axe is usually near to hand. It will soon be hung on the wall. Not because I am preparing for the zombie apocalypse, but because it is both useful as a tool, and as a visual reminder about what it means to build quality products. If this ramble of mine isn’t enough of a distraction, watch Why Values are Important to understand how this axe relates to measures of quality especially in product design.
This toaster is also wonderful. We’ve had this toaster more than 10 years now, and it works perfectly. If it were to break, I can get the parts locally and service it myself (it’s deliberately built to last and be repaired). It was an expensive initial purchase, but works out cheap in the long run. If it broke today, I would fix it. If I couldn’t fix it for some extreme reason, I would buy the same toaster in a blink. It is a high quality product.
This is the espresso coffee I drink every day. Not the tin, it’s another brand that comes in a bag. It has been consistently good for a couple of years until the last two weeks when the grind has been finer than usual and it keeps blocking the machine. It was a high-quality product in my mind, until recently. I’ll let another batch pass through the supermarket shelves and try it again. Otherwise I’ll switch.
This spatula looks like a novelty product and typically I don’t think very much of novelty products in place of useful tools, but it’s actually a high quality product. It was a gift, and we use it a lot and it just works really well. If it went missing today, I’d want to get another one the same. Saying that, it’s surprisingly expensive for a spatula. I’ve only just looked at the price, as a result of writing this. I think I’d pay that price though.
All of those examples are relatively expensive products within their respective categories, but price is not the measure of quality, even if price sometimes correlates with quality. I’ll get on to this.
How about things of quality that are not expensive in this way?
What is quality music, or art, or literature to you? Is it something new you enjoy today? Or something you enjoyed several years ago? I personally think it’s the combination of those two things. And I posit that you can’t know the real quality of something until enough time has passed. Though ‘enough time’ varies by product.
Ten years ago, I thought all the music I listened to was of high quality. Re-listening today, I think some of it was high-quality. As an exercise, listen to some music you haven’t for a while, and think about which tracks you enjoy for the nostalgia and which you enjoy for the music itself.
In the past, we had to rely on sales as a measure of the popularity of music. But like price, sales doesn’t always relate to quality. Initial popularity indicates potential quality, but not quality in itself (or it indicates manipulation of the audience via effective marketing). Though there are debates around streaming music services and artist payment, we do now have data points about the ongoing value of music beyond the initial parting of listener from cash. I think this can do interesting things for the quality of music overall. And in particular that the future is bleak for album filler tracks when you’re paid per stream.
Another question I enjoy thinking about is why over the centuries, some art has lasting value, and other art doesn’t. But I think I’ve taken enough tangents for now.
So, to join this up.
My view is that quality is reflected by loyalty. And for most products and services, end-user loyalty is something you can measure and optimize for.
Loyalty comes from building things that both last, and continue to be used.
Every other measurable detail about quality adds up to that.
Reducing the defect rate of component X by 10% doesn’t matter unless it impacts on the end-user loyalty.
It’s harder to measure, but this is true even for things which are specifically designed not to last. In particular, “experiences”; a once-in-a-lifetime trip, a festival, a learning experience, etc, etc. If these experiences are of high quality, the memory lasts and you re-live them and re-use them many times over. You tell stories of the experience and you refer your friends. You are loyal to the experience.
Bringing this back to work.
For MoFo colleagues reading this, our organization goals this year already point us towards Quality. We use the industry term ‘Retention’. We have targets for Retention Rates and Ongoing Teaching Activity (i.e. retained teachers). And while the word ‘retention’ sounds a bit cold and business like, it’s really the same thing as measuring ‘loyalty’. I like the word loyalty but people have different views about it (in particular whether it’s earned or expected).
This overarching theme also aligns nicely with the overall Mozilla goal of increasing the ‘number of long term relationships’ we hold with our users.
Language is interesting though. Thinking about a ‘20% user loyalty rate’ 7 days after sign-up focuses my mind slightly differently than a ‘20% retention rate’. ‘Retention’ can sound a bit too much like ‘detention’, which might explain why so many businesses strive for consumer ‘lock-in’ as part of their business model.
Talking to OpenMatt about this recently he put a better MoFo frame on it than loyalty; Retention is a measure of how much people love what we’re doing. When we set goals for increasing retention rate, we are committing to building things people love so much that they keep coming back for more.
- You can measure quality by measuring loyalty
- I’m happy retention rates are one of our KPIs this year
My next post will look more specifically about the numbers and how retention rates factor into product growth.
And I’ll try not to make it another essay. 😉
What’s happening this week?
My number one goal (P1) for this week is solving offline friendly mobile analytics for Webmaker App, while keeping other projects ticking along adequately.
Here’s to a productive week.
One of this week’s conversations was with Nesta, about Webmaker usage within the UK and whether or not we have data to support the theory that face-t0-face events have an impact getting people involved in making on the web. These are two topics that interest me greatly.
I’m basically copying some of my notes into blog form so that the conversation isn’t confined to a few in-boxes.
And the TL;DR is our data represents what we’ve done, rather than any universal truth.
Our current data would support the hypothesis that face-to-face time is important for learning, but that would simply be because that’s how our program has been designed to date. In other words, our Webmaker tools were designed primarily for use in face-to-face events, which meant that adoption by ‘self-learners’ online is low because their is little guidance or motivation to play with our tools on your own. This year we’re making a stronger push on developing tools that can be used remotely, alongside our work on volunteer led face-to-face events. This will lead to a less biased overall data set in the future where we can begin to properly explore the impact on making and learning for people who do or don’t attend face-to-face events at various stages in their learning experience. In particular I’m keen to understand what factors help people transition from learners, to mentoring and supporting their peers.
I also took a quick look at the aggregate Google Analytics location data for the UK audience which I hadn’t done before and which re-enforces the point above.
Above: Traffic to Webmaker (loosely indicating an interest in the topic) is roughly distributed like a population map of the UK. This is what I expect to see of most location data.
Above: However, if you look at the locations of visitors who make something, there are lots of clusters around the UK and London is equaled by many other cities.
To-date, usage of the Webmaker tools has been driven by those who are using the tools to teach the web (i.e. Webmaker Mentors). But we also know there are large numbers of people who find Webmaker outside of the face-to-face event scenarios who need a better route into Webmaker’s offering.
The good news is that this year’s plans look after both sets of potential learners.
I wrote a post over on fundraising.mozilla.org about our latest round of optimization work for our End of Year Fundraising campaign.
We’ve been sprinting on this during the Mozilla all-hands workweek in Portland, which has been a lot of fun working face-to-face with the awesome team making this happen.
You can follow along with the campaign, and see how were doing at fundraising.mozilla.org
And of course, we’d be over the moon if you wanted to make a donation.
If I find a moment, I’ll write about many of the fun and inspiring things I saw at Mozfest this weekend, but this post is about a single session I had the pleasure of hosting alongside Andrew, Doug and Simon; Learning Analytics for Good in the Age of Big Data.
We had an hour, no idea if anyone else would be interested, or what angle people would come to the session from. And given that, I think it worked out pretty well.
We had about 20 participants, and broke into four groups to talk about Learning Analytics from roughly 3 starting points (though all the discussions overlapped):
- Practical solutions to measuring learning as it happens online
- The ethical complications of tracking (even when you want to optimise for something positive – e.g. Learning)
- The research opportunities for publishing and connecting learning data
But, did anyone learn anything in our Learning Analytics session?
Well, I know for sure the answer is yes… as I personally learned things. But did anyone else?
I spoke to people later in the day who told me they learned things. Is that good enough?
As I watched the group during the session I saw conversations that bounced back and forth in a way that rarely happens without people learning something. But how does anyone else who wasn’t there know if our session had an impact?
How much did people learn?
This is essentially the challenge of Learning Analytics. And I did give this some thought before the session…
As a meta-exercise, everyone who attended the session had a question to answer at the start and end. We also gave them a place to write their email address and to link their ‘learning data’ to them in an identifiable way. It was a little bit silly, but it was something to think about.
This isn’t good science, but it tells a story. And I hope it was a useful cue for the people joining the session.
- We had about 20 participants
- 10 returned the survey (i.e. opted in to ‘tracking’), by answering question 1
- 5 of those answered question 2
- 5 gave their email address (not exactly the same 5 who answered both questions)
Here is our Learning Analytics data from our session
Is that demonstrable impact?
Even though this wasn’t a serious exercise. I think we can confidently argue that some people did learn, in much the same way certain newspapers can make a headline out of two data points…
What, and how much they learned, and if it will be useful later in their life is another matter.
Even with the deliberate choice of question which was almost impossible to not show improvement from start to end of the session, one respondent claims to be less sure what the session was about after attending (but let’s not dwell on that!).
Post-it notes and scribbles
If you were at the session, and want to jog your memory about what we talked about. I kind-of documented the various things we captured on paper.
I’m looking forward to exploring Learning Analytics in the context of Webmaker much more in 2015.
And to think that this was just one hour in a weekend full of the kinds of conversations that repeat in your mind all the way until next Mozfest. It’s exhausting in the best possible way.
I’m back at the screen after a week of paternity leave, and I’ll be working part-time for next two weeks while we settle in to the new family routine at home.
In the meantime, I wanted to mention a Mozilla contributor analysis project in case people would like to get involved.
We have a wiki page now, which means it’s a real thing. And here are some words my sleep-deprived brain prepared for you earlier today:
The goal and scope of the work:
Explore existing contribution datasets to look for possible insights and metrics that would be useful to monitor on an ongoing basis, before the co-incident workweek in Portland at the beginning of December.
- Stress-test our current capacity to use existing contribution data
- Look for actionable insights to support Mozilla-wide community building efforts
- Run ad-hoc analysis before building any ‘tools’
- If useful, prototype tools that can be re-used for ongoing insights into community health
- Build processes so that contributors can get involved in this metrics work
- Document gaps in our existing data / knowledge
- Document ideas for future analysis and exploration
I’m very excited that three members of the community have already offered to support the project and we’ve barely even started.
In the end, these numbers we’re looking at are about the community, and for the benefit of the community, so the more community involvement there is in this process, the better.
If you’re interested in data analysis, or know someone who is, send them the link.
This project is one of my priorities over the following 4-8 weeks. On that note, this looks quite appealing right now.
So I’m going make more tea and eat more biscuits.
- Removing the second sentence increases conversion rate (hypothesis = simplicity is good).
- The button text ‘Go!’ increased the conversion rate.
- Both variations on the headline increased conversion rate, but ‘Welcome to Webmaker’ performed the best.
- We should remove the bullet points on this landing page.
- The log-in option is useful on the page, even for a cold audience who we assume do not have accounts already.
- Repeating the ask ‘Sign-up for Webmaker’ at the end of the copy, even when it duplicates the heading immediately above, is useful. Even at the expense of making the copy longer.
- The button text ‘Create an account’ works better than ‘Sign up for Webmaker’ even when the headline and CTA in the copy are ‘Sign up for Webmaker’.
- These two headlines are equivalent. In the absence of other data we should keep the version which includes the brand name, as it adds one further ‘brand impression’ to the user journey.
- The existing blue background color is the best variant, given the rest of the page right now.
The Webmaker Testing Hub
If any of those “conclusions” sound interesting to you, you’ll probably want to read more about them on the Webmaker Testing Hub (it’s a fancy name for a list on a wiki).
This is where we’ll try and share the results of any test we run, and document the tests currently running.
And why that image for this blog post?
Because blog posts need and image, and this song came on as I was writing it. And I’m sure it’s a song about statistical significance, or counting, or something…