Measuring Quality

At the end of last year, Cassie raised the question of ‘how to measure quality?’ on our metrics mailing list, which is an excellent question. And like the best questions, I come back to it often. So, I figured it needed a blog post.

There are a bunch of tactical opportunities to measure quality in various processes, like the QA data you might extract from a production line for example. And while those details interest me, this thought process always bubbles up to the aggregate concept: what’s a consistent measure of quality across any product or service?

I have a short answer, but while you’re here I’ll walk you through how I get there. Including some examples of things I think are of high quality.

One of the reasons this question is interesting, is that it’s quite common to divide up data into quantitative and qualitative buckets. Often splitting the crisp metrics we use as our KPIs from the things we think indicate real quality. But, if you care about quality, and you operate at ‘scale’, you need a quantitative measure of quality.

On that note, in a small business or on a small project, the quality feedback loop is often direct to the people making design decisions that affect quality. You can look at the customers in your bakery and get a feel for the quality of your business and products. This is why small initiatives are sometimes immensely high in quality but then deteriorate as they attempt to replicate and scale what they do.

What I’m thinking about here is how to measure quality at scale.

Some things of quality, IMHO:

axeThis axe is wonderful. As my office is also my workshop, this axe is usually near to hand. It will soon be hung on the wall. Not because I am preparing for the zombie apocalypse, but because it is both useful as a tool, and as a visual reminder about what it means to build quality products. If this ramble of mine isn’t enough of a distraction, watch Why Values are Important to understand how this axe relates to measures of quality especially in product design.

toasterThis toaster is also wonderful. We’ve had this toaster more than 10 years now, and it works perfectly. If it were to break, I can get the parts locally and service it myself (it’s deliberately built to last and be repaired). It was an expensive initial purchase, but works out cheap in the long run. If it broke today, I would fix it. If I couldn’t fix it for some extreme reason, I would buy the same toaster in a blink. It is a high quality product.

coffeeThis is the espresso coffee I drink every day. Not the tin, it’s another brand that comes in a bag. It has been consistently good for a couple of years until the last two weeks when the grind has been finer than usual and it keeps blocking the machine. It was a high-quality product in my mind, until recently. I’ll let another batch pass through the supermarket shelves and try it again. Otherwise I’ll switch.

spatulaThis spatula looks like a novelty product and typically I don’t think very much of novelty products in place of useful tools, but it’s actually a high quality product. It was a gift, and we use it a lot and it just works really well. If it went missing today, I’d want to get another one the same. Saying that, it’s surprisingly expensive for a spatula. I’ve only just looked at the price, as a result of writing this. I think I’d pay that price though.

All of those examples are relatively expensive products within their respective categories, but price is not the measure of quality, even if price sometimes correlates with quality. I’ll get on to this.

How about things of quality that are not expensive in this way?

What is quality music, or art, or literature to you? Is it something new you enjoy today? Or something you enjoyed several years ago? I personally think it’s the combination of those two things. And I posit that you can’t know the real quality of something until enough time has passed. Though ‘enough time’ varies by product.

Ten years ago, I thought all the music I listened to was of high quality. Re-listening today, I think some of it was high-quality. As an exercise, listen to some music you haven’t for a while, and think about which tracks you enjoy for the nostalgia and which you enjoy for the music itself.

In the past, we had to rely on sales as a measure of the popularity of music. But like price, sales doesn’t always relate to quality. Initial popularity indicates potential quality, but not quality in itself (or it indicates manipulation of the audience via effective marketing). Though there are debates around streaming music services and artist payment, we do now have data points about the ongoing value of music beyond the initial parting of listener from cash. I think this can do interesting things for the quality of music overall. And in particular that the future is bleak for album filler tracks when you’re paid per stream.

Another question I enjoy thinking about is why over the centuries, some art has lasting value, and other art doesn’t. But I think I’ve taken enough tangents for now.

So, to join this up.

My view is that quality is reflected by loyalty. And for most products and services, end-user loyalty is something you can measure and optimize for.

Loyalty comes from building things that both last, and continue to be used.

Every other measurable detail about quality adds up to that.

Reducing the defect rate of component X by 10% doesn’t matter unless it impacts on the end-user loyalty.

It’s harder to measure, but this is true even for things which are specifically designed not to last. In particular, “experiences”; a once-in-a-lifetime trip, a festival, a learning experience, etc, etc. If these experiences are of high quality, the memory lasts and you re-live them and re-use them many times over. You tell stories of the experience and you refer your friends. You are loyal to the experience.

Bringing this back to work.

For MoFo colleagues reading this, our organization goals this year already point us towards Quality. We use the industry term ‘Retention’. We have targets for Retention Rates and Ongoing Teaching Activity (i.e. retained teachers). And while the word ‘retention’ sounds a bit cold and business like, it’s really the same thing as measuring ‘loyalty’. I like the word loyalty but people have different views about it (in particular whether it’s earned or expected).

This overarching theme also aligns nicely with the overall Mozilla goal of increasing the ‘number of long term relationships’ we hold with our users.

Language is interesting though. Thinking about a ‘20% user loyalty rate’ 7 days after sign-up focuses my mind slightly differently than a ‘20% retention rate’. ‘Retention’ can sound a bit too much like ‘detention’, which might explain why so many businesses strive for consumer ‘lock-in’ as part of their business model.

Talking to OpenMatt about this recently he put a better MoFo frame on it than loyalty; Retention is a measure of how much people love what we’re doing. When we set goals for increasing retention rate, we are committing to building things people love so much that they keep coming back for more.

In summary:

  • You can measure quality by measuring loyalty
  • I’m happy retention rates are one of our KPIs this year

My next post will look more specifically about the numbers and how retention rates factor into product growth.

And I’ll try not to make it another essay. ūüėČ

Remembering maps from memory

Today, I found this awesome post on Uncertain Cartographies (via Flowing Data), and it immediately took me back to something I made when I was in college and studying fine art.

So check out that link first, as this post will make more sense in relation to it, and it’s pretty fascinating anyway.

Then I’ll continue my reminiscence… :)

I used to have a framed print of a map I’d drawn on the wall at home, though when I say “print” it was ~10 bits of A4 photo paper I had carefully cut and glued together. It lasted about 6 years on the wall before the ink faded and the paper peeled and I had to take it down. And I hadn’t thought about this map again until today.

Then after reading this paragraph in particular, I really wanted to find my old map:

“Places where I once lived are deeply etched in my mind. Given a blank sheet of paper, and a little lenience, I can draw a respectable map of Murrays Bay, Mount Eden, Kingsland, Longburn, Summer Hill or Mana from memory. Yet most New Zealand localities are at once familiar and largely unknown to me.”

I went searching for my old map.

Amazingly (to me at least) I managed to find a copy of my original Photoshop artwork file on an old archive CD-R that’s followed me from desk to desk, between many house moves and a couple of countries and that I’ve not had any cause to look at until today.

I’ve posted the map below, but first here’s some context:

This was something I created 10 years ago when I was 18 and I’d been driving for less than a year. I didn’t really use maps for directions, but I drove quite a lot to play gigs with my band, and to visit my girlfriend (now wife) who was studying in Exeter. I knew my way around by which junctions led to where, but I’d never had to ‘map’ these places in a proper geographic context in my mind. This is pre-Google Maps/Earth (hence the AA roadmap styling) and it’s interesting now to think how the ability to browse maps so quickly and fluidly online shapes the way we visualize the abstraction of location in our minds. Even more so when you factor in navigation aids.

To create the map, I sat down with a pen and a large piece of paper and drew the roads leading out around me to all the places I had traveled in recent months. I didn’t allow edits or corrections, and there was no advanced sketching of where places were located. I drew the roads and made the links between places in one go. I did this until I ran out of roads I could meaningfully label. Scanned the whole thing in and traced the lines in Photoshop (no fixes allowed). I recall this “giant” file crashing my computer on a regular basis, though looking at it now, it’s only 50mb.

Anyway, that’s getting a bit nostalgic and I don’t mean it to. What I meant to do was post this map as a response to the thought provoking piece I read today.

A little insight into my earlier years
A little insight into my earlier years



Available hours in a year for personal projects

While planning ahead to finish my Open University studies, I’ve been testing how well I can study in my available free time; and my recent study with Coursera has provided a pretty good simulation.

It’s important to be realistic with yourself about how much time you actually have to do these things, on a sustainable basis, for a significant period of time. Especially with the tuition fees being as expensive as they are and if you’re making a commitment for a whole year of your free time.

My thinking has gone like this…

First I account for my time being a husband and dad, then my working hours, then sleep, then a few hours for getting/keeping fit and I’m left with around two hours per day, or 14 hours a week of ‘free’ time. For a couple of weeks at a time, it is possible to fill those 14 hours completely with study or to make progress on a project, but it’s exhausting, and over a longer period like a year, it just won’t work. In those 14 free hours I need some downtime. I need at least a couple of nights off to watch a film, kill some aliens in a computer game or enjoy a good single malt. If you don’t plan for downtime, you’re not being realistic and you will be less effective.

This is not an issue of direction, but of how much fuel is in the tank.

After my tests and calculations, what I’m left with is about nine hours each week for stretching myself with new things (there are new things to do in my working hours too, but that’s not quite the same).

I have many many lists of things I want to build, write, make, test, learn and do, so coming to terms with the finite number of hours available in a week, and therefore a year, is always a battle with myself. But it’s an important battle and I’m in one my more realistic planning phases right now.

Bringing this thought back to my studies, finishing my degree has become more complex than I first expected because my route to this current point doesn’t fit into the standard institutional boxes, which now excludes me from a student loan. I also need to choose modules that work with the time I can commit so I’ve been working through a few spreadsheets to make sense of my options. We’ll see what happens, but it might be I can’t afford a degree ‘with honours’, but I can live with that. It’ll still be a BSc, largely in computer science with a chunk of something random at the end. It suits me quite well.

My study will start again proper in October, so I have nine weeks left to fit in a final personal project for this year. All other ideas must be put on ice until late 2014.

Nine weeks is about 81 hours.

I had been toying with the idea of building a game of some sort, as my recent messing about was enjoyable, but after watching Indie Game The Movie (I recommend it by the way), I realise how much of an overcommitment that would be for 81 hours. Even a simple game would be unachievable in that time given the number of new tools I’d be learning in the process. All game ideas are frozen.

So I have another idea, one that should fit in the time.

But I’ll give it some more thought before talking about it here.

One parting thought for this evening.

Nine hours a week might not sound like much, but it adds up.

468 hours in a year is about the equivalent of three months of full-time work. That’s a useful reference point when planning what you want to get done in the next year.

You need a process to work effectively with so many small chunks of time, but think of what you could do with a quarter of a year of working time.

If you have an idea.

If you think you could do it in three months of regular working hours.

Get making.

This isn’t so scary:

3 x 2hr weeknight sessions
1 x 3hr weekend session

Degrees of done

With life in a reasonably calm and sensible place right now, it seems like a good time to finish up something I started some time ago.

About 10 years ago, when I finished art college I came to the conclusion that going to university would lead to a big old pile of debt, and that I could find a better way to navigate the requirements of professional life.

My studies were in fine art which was a useful¬†exercise¬†in creative and critical thinking, but was never going to pay the bills. And I’m not planning to die a starving artist. Alongside my studies, I’d been building websites (and earning a few pounds doing it). I’d learned enough about writing code that I wanted to study computer science (CS) more formally, but even a distinction in fine art wouldn’t get¬†me¬†into any regular university courses in computer science.

Academically, and even now with professional job opportunities, the venn diagram of visual art and computer science rarely overlaps. I think this needs to change, and probably will. Code and composition are just tools and the more closely they work together the better.

So anyway, the solution I found to my dilemma of not being ‘qualified’ to study for the qualification I wanted, and not being happy with the cost of getting it either, was to study computer science with¬†the Open University (OU).

I was an edge-case OU applicant back then, as very few people at ‘standard’ university age considered the OU¬†as an option. Their students were typically older working professionals or the recently retired.¬†The choice for my peers was typically between a bricks-and-mortar university or to start work. Via the OU I was able to study and start work, avoid any debt and get some significant professional experience at the same time. And by typical graduation age, I had been happily employed for some time. Sometimes I’d get up at 4am to study, other times I’d work through the night on weekends, but I enjoyed it.

And all in all, I’m still very pleased with the choice that the young me made.

Over four years of studying while working, I completed two thirds of a CS degree, mostly using coding skills I’d actually been learning on the job, and then in 2008 I decided to take a year off. In that year-off from study I took a new job working at WWF and have had enough personal projects and general life things to keep me busy in the five years since. The degree is unfinished business, and I’m thinking about finishing it now.

The thing is, I don’t need this degree to get a job (I have a dream job already), but I don’t like leaving it undone. And it’s been bugging me.

So I’ve been looking at what courses are left to finish my degree and I discovered a couple of things.

  1. The OU have swapped around the modules for their courses while I’ve been absent, so I no longer have two thirds of a CS degree. I instead have about¬†a sixth of many of their¬†new CS based degrees. I do not want to do another five sixths of a CS degree. It’s not bugging me that much.
  2. Dave and Nick have been screwing around, and the last third of my degree is going to cost 50% more than the first two thirds of my degree combined. Damn you Dave and Nick.

But there is some good news. For now at least, the OU offer a slightly¬†unusual degree called simply¬†an Open Degree. You can study modules in any topic so long as you have the right number of credits from each level of study to add up to a regular honours degree’s worth of qualification (either BSc or BA). So while my original CS degree has gone out the window, I still have two thirds of an Open Degree.

My plan now is to complete a few more modules in the next couple of years and finish off my degree, even if it’s not the one I started. The “openness” of the Open Degree also allows me to diversify my study a bit, rather than repeat the skills I already use professionally, and that makes it more interesting. I’m particularly looking forward to a course on¬†modeling¬†ecosystems.

The downside to this plan is that the recent increases in UK tuition fees may now require me to look into a student loan after all. Again, damn you Dave and Nick. This is going to be an expensive itch to scratch.

Anyway, that’s a long way of saying I’ll be doing some studying later this year.

Saying that, I’m doing some studying now so that’s not really news. And on the aforementioned issue of the computer science and visual art venn diagram, the overlap is captured wonderfully in this Coursera course that’s currently running:¬†¬†“Creative Programming for Digital Media & Mobile Apps”¬†

This course has been so enjoyable, that it’s partly to¬†blame for the expensive year now ahead. Damn¬†no,¬†thank you Coursera. Damn you Dave and Nick.

For a free and open internet, be quick

“On December 3rd, the world‚Äôs governments will meet to update a key treaty of a UN agency called the International Telecommunication Union (ITU). Some governments are proposing to extend ITU authority to Internet governance in ways that could threaten Internet openness and innovation, increase access costs, and erode human rights online.” – src:¬†

Here are a couple of places you can show your support for a free and open web right away:

If you have a bit more time, you can get creative with Mozilla’s Webmaker kit

You can see who is speaking on your behalf here:

And this article sums up the transparency issues:

On 3D printing, and a world unready

With the recent release of the Makerbot Replicator 2, 3D printing tipped into the real world. It moved from a conceptual idea that geeks and tinkerers would try and explain to their doubtful loved ones into something you can have delivered to your door in a matter of working days.

It changes everything, and I think the world will be caught off guard.

Even now, the ~$2,000 price tag isn’t crazy. It’s more than I would pay, but plenty can afford it. And this time next year the price will halve, the resolution will double and it will print slightly bigger things. Same again the following year. In four or five years’ time, almost anyone who wants one will be able to justify the cost of a 3D printer to¬†themselves.

I always imagined playing with Lego with my son when he’s older, but it might be we’re printing our own Lego-esque blocks too.

I can only begin to imagine what this will do to a business like Lego. Sure, it costs you more to print a block than it costs Lego to print a block, but when you pay ¬£10 for a box of 25 Lego pieces you’re not paying for material; you’re paying for branding, marketing, packaging, tax, distribution and their profit. Printing your own blocks will¬†definitely¬†be cheaper for the end consumer and anyone could learn to model a number of Lego blocks in a 3D modelling tool like Sketchup in less than 4 hours.

I had a quick think about the kinds of businesses who sell cheap-to-make, but expensive-to-sell plastic products whose profit margin is primarily the relationship between their business running costs and the perceived value of the end product. I’m not saying there is no merit to the price we pay for these products today, but we’re¬†definitely not paying manufacturing cost plus ~20%.

3D printing changes everything for businesses like these:

  • Lego
  • Tupperware
  • Airfix
  • People who sell spare parts for things like old washing machines
  • Games Workshop
  • People like Shapeways who sell 3D printing as a service (they’ll soon start to look like photocopying shops do today)
  • And there will be many more

Broke your ladle while cooking? Don’t worry, you can print a new one before you need to serve dinner.

It will almost always be cheaper to ship a solid reel of plastic in it’s compact form than the same amount of plastic in its final product form full of holes and empty spaces.


Gorilla Pod and Similar MakerBot tripod

In a few years time, will you buy a Gorilla Pod and wait the tedious ~18hrs for Amazon Prime next day delivery (how slow and old fashioned!) or will you print one out right then and there?

While business is a start, a lens through which we think about the future, 3D printing has implications everywhere in this world.

There’s already been plenty of coverage about the ability to print your own weapons. While you can take the plans off of public websites, the digital files are so easy to share that you can’t take them out of circulation. Weapons may well become less regulated (and¬†regulate-able) then they are today.

In politics, what good are trade sanctions when you can move goods digitally across borders for production locally? Hyper-locally even. This is now impossible to stop. We have to re-think international trade.

In development, what does this mean for people who live in rural Kenya. Do aid organizations deliver end products they think will be useful, or spools of printing plastic to people in need?

Maybe we’ll print our own artificial limbs?

So what skills do we need to be developing? 

I don’t think we all need to be taking a course in CAD, because¬†it’s so easy to share 3D files. They’re smaller than MP3s and much smaller than videos and we don’t seem to have any trouble moving these around the world. A few thousand skilled contributors will produce the bulk of the designs used by the billions of people around the planet. Look at all things you can already download and print today.

What’s the future for the businesses affected?

If I ran Lego, I’d be thinking about how to phase out manufacturing while retaining brand and thought leadership. Not immediately, but probably within five years, ten at most. I’d focus on selling amazing plans for kits and give up on trying to own the right to the blocks themselves.¬†The overall size of the business would shrink, but it might be possible to maintain the profits. Selling plans digitally would hugely cut costs and by¬†embracing¬†this new world, Lego can be a relevant and meaningful part of it. Most likely though, they’ll¬†unleash¬†a team of lawyers on any website that offers 3D models of anything that looks like Lego. People will print their own blocks anyway, sales will decline and there will be plenty of negative press coverage about the company who tries to sue the 10 year old girl who made her own Lego-like designs for her friends birthday. I don’t say this because I think that’s a Lego like trait, but because it’s the way all large corporations today tend to behave when they come under threat.

In many ways, the future will be more different than we can possibly imagine.

Interwoven with bits

The Internet, as a malleable collection of bits, responds to and evolves with the cumulative needs of the connected human race. It rewards, condemns or ignores elements of itself in line with our primitive human desires; growing in places and dying in others. So we, as human beings with our natural, non-digital desires, shape the Internet.

Simultaneously, the Internet changes how we (cumulatively at least) behave. It creates and destroys jobs, it makes and breaks marriages, and it redefines where, how and with who we live and share our lives. And at the deepest level it rewires the chemical makeup of our brains by changing the way we think, speak and remember. While it divides two generations, it keeps them in touch.

We and the Internet are now dependent on one other; we are interwoven with bits. As we shape it, it shapes us.

Just as we are interwoven with books, advertising, commerce, tools, religion, industry, government, agriculture, transport and much much more.

This is not something to be afraid of, but it’s something worth trying to understand.

On wanting to conquer the world

The Other F WordI’ve spent the last couple of weeks watching updates come in from my old band as they tour around Europe with some of my teenage heroes. It’s a bittersweet experience. It’s really great to see the band going strong, but I can’t pretend I’m not a little bit jealous. Teenage me would definitely give adult me a hard time about this if I went back to meet him. Especially if I met him as he’s watching Reel Big Fish at Reading Festival. He’d tell adult me that one day he’d support RBF on tour, and he knew it could happen. He was right, damn him. It would just take about eight years longer than he expected, by which time he’d be doing something very different with his life.

Now, I’m really not complaining. I was very lucky. I met and gigged with many of my heroes, and most of my heroes turned out to be lovely people, which is awesome. But, there is a part of me that will always remember my teenage vision of life on the road in a punk-rock band; a life I only 80% realised before I moved on. That will always be a loose end I carry with me. But better a loose end than no thread at all. Shepherd’s Bush would have been fun though.

Anyway, the point of this post wasn’t strictly to reminisce, it’s mainly about the future. A future where anytime now I could become a father (11 days until d-day!).

Understandably (I hope), I’ve been thinking about the choices I’ve made in this life and the road I’ve walked so far. And as I think it’s healthy to remember teenage dreams, I was asking myself something along these lines:

  • How would fatherhood fit with my teenage dream of punk-rock life on the road?
  • How do you challenge the system of the world, while still living in it?

So Pheebs and I watched The Other F Word. Who would have thought that someone would make a documentary dealing with exactly those questions? It’s by no means the best film ever made, but it was definitely the most weirdly specific and well timed documentary I’ve ever watched. I’d recommend it if you’re interested in exactly the same questions, and you happened to like 90s punk and you’re about to become a father.

I found my answer somewhere just off the edge of the screen, in a song that wasn’t featured in the film. Somewhere near the start, there was an acoustic version of Bad Religion’s Sorrow, and I was left with a lyric in my head that lasted the whole film. It was actually from another Bad Religion song I Want to Conquer the World, though I only twigged that later:

“Hey Mr. Diplomat with your worldly aspirations, did you see your children cry when you left them at the station?”

The words crept out from my teenage memory and stayed with me as I watched Jim Lindberg trying to Skype with his daughters after something like 200 days on the road. The hotel connection was poor, and we’re left watching the girls talk to the black void where their father’s face used to be. They don’t even know he has gone.

This life will always be a paradox. At best I can contemplate the poles.

What separates a diplomat and a punk-rocker?

Sell out?