2015 Mozilla Foundation Metrics Strategy(ish) & Roadmap(ish)

I wrote a version of this strategy in January but hadn’t published it as I was trying to remove those ‘ish‘s from the title. But the ‘ish’ is actually a big part of my day-to-day work, so this version embraces the ‘ish’.

MoFo Metrics Measures of Success:

These are ironically, more qualitative than quantitative.

  1. Every contributor (paid or volunteer) knows at any given time what number they (or we) are trying to move, where that number is right now, and how they hope to influence it.
  2. We consider metrics (i.e. measures of success) before, during and after after each project.
  3. We articulate the stories behind the metrics we aim for, so their relevance isn’t lost in the numbers.
  4. A/B style testing practice has a significant impact on the performance of our ‘mass audience’ products and campaigns.

1. Every contributor (paid or volunteer) knows at any given time what number they (or we) are trying to move, where that number is right now, and how they hope to influence it.

  • “Every” is ambitious, but it sets the right tone.
  • This includes:
    • Public dashboards, like those at https://metrics.webmaker.org
    • Updates and storytelling throughout the year
    • Building feedback loops between the process, the work and the results (the impact)

2. We consider metrics (i.e. measures of success) before, during and after after each piece of work.

  • This requires close integration into our organizational planning process
  • This work is underway, but it will take time (and many repetitions) before it becomes habit

3. We articulate the stories behind the metrics we aim for, so their relevance isn’t lost in the numbers.

  • The numbers should be for navigation, rather than fuel

4. A/B style testing practice has a significant impact on the performance of our ‘mass audience’ products and campaigns.

  • This is the growth hacking part of the plan
  • We’ve had some successes (e.g. Webmaker and Fundraising)
  • This needs to become a continuous process

Those are my goals.

In many cases, the ultimate measure of success is when this work is done by the team rather than by me for the team.

We’re working on Process AND Culture

Process and culture feed off of and influence each other. Processes must suit the culture being cultivated. A data driven culture can blinker creativity – it doesn’t have to, but it can. And a culture that doesn’t care for data, won’t care for processes related to data. This strategy aims to balance the needs of both.

A roadmap?

I tried to write one, but basically this strategy will respond to the roadmaps of each of the MoFo teams.

So, what does Metrics work look like in 2015?

  • Building the tools and dashboards to provide the organisational visibility we need for our KPIs
  • ‘Instrumenting’ our products so that we can accurately measure how they are being used
  • Running Optimization experiments against high profile campaigns
  • Running training and support for Google Analytics, Optimizely, and other tools
  • Running project level reporting and analysis to support iterative development
  • Consulting to the Community Development Team to plan experimental initiatives

Plus: supporting teams to implement our data practices, and of course, the unknown unknowns.

…ish

A ‘free’ online learning experience

2862656849_f0fa5c78bf_oI’ve blogged about various experiences of online learning I’ve taken part in over the years and wanted to reflect on the most recent one. Coursera’s three week Introduction to Ableton Live.

Learning more about learning is one of my personal goals this year. And I find writing out loud to be useful tool in thinking. So that’s mostly the point of this.

I take these courses mostly because I like learning new things, but also because I’m interested in online learning more generally. How do you most effectively transfer knowledge, skills and motivation via the web, and/or about the web? That question is often on my mind.

Almost all of the projects I work on at Mozilla are somewhere in the education space; directly with Webmaker or Mozilla Learning Networks and tangentially in the topic of volunteer contribution. Contributing to an open source project as complex and distributed as Mozilla is a learning experience in itself, and sometimes requires specific training to even make it possible.

To further frame this particular brain dump, I’m also interested generally in the economics of the web and how this shapes user experiences, and I have strong feelings about the impact of advertising’s underlying messaging and what this does over-time when it dominates a person’s daily content intake. I’m generally wary of the word “Free”. This all gets complex when you work on the web, and even directly on advertising at times. Most of my paycheques have had some pretty direct link to the advertising world, except maybe when I was serving school dinners to very rich children – but that wasn’t my favourite job, despite it’s lack of direct societal quandaries.

Now, to the content…

If you’re like me, you will tend to read notes about a topic like ‘commerce in education’ and react negatively to some of these observations because there are many cases where those two things should be kept as far apart as possible. But I’m actually not trying to say anything negative here. These are just observations.

Observations

All roads lead to… $

$ Coursera

My online experience within the Coursera site was regularly interrupted with a modal (think popup) screen asking if I wanted to pay to enrol in the ‘Signature Track’, and get a more official certification. This is Coursera’s business model and understandably their interest. It wasn’t at all relevant to me in my life situation, as I was taking a course about how to play with fun music software in my free time. I don’t often check my own qualifications before I let myself hobby. Not that anyone checked my qualifications before they let me work either, but I digress. Coursera’s tagline says ‘free’, but they want you to pay.

$ Blend.io

All assignments for the course had to be published to Blend for peer-evalutation, Blend is like Github but for raw audio production tracks rather than source-code. I didn’t know about Blend before the course, and really like it as a concept and how it’s executed and for what it could do for collaborative music making. But I note, it is a business. This course funnels tens of thousands of new users into that business over the course of a few days. There might not be any direct financial trade here (between companies for example), but users are capital in start-up land. And I now receive emails from Blend with advertisements for commercial audio production tools. My eyeballs, like yours, have a value.

$ Berklee College of Music

While hosted on Coursera, the content of this course is by Berklee College of Music. The content they ‘give away’ would traditionally only have been available to paying students. Berklee’s business is selling seats in classes. This course isn’t given away as an act of kindness, it’s marketing. Three weeks is short and therefore the content is ‘light’. Lighter than I was expecting (not that I’m entitled). But halfway through, we receive a promotional email about Berklee’s own online education platform where you could create an account to get access to further ‘free’ videos to supplement the Coursera materials. I found these supplementary videos more useful, and they lead to offers to sign-up for extended paid courses with Berklee Online. For Berklee, this whole excercise is a marketing funnel. Quite possibly it’s the most fun and least offensive marketing funnel you can be dropped into, but it exists to do that job.

$ Erin Barra – Course professor and artist

Now, I write this with genuine sympathy, as I’ve walked the floor at countless venues trying to sell enough music and merch to cover the petrol costs of playing a gig. But this is a commercial element of this learning experience, so I will note it. At many points throughout the three weeks, we had opportunities to buy Erin’s music, t-shirts, and audio production stems (these are like a layer file of an original recording) for consumption and or remixing. I know you have to hustle if you’re making music for a living, but the observation here is that the students of this course are also a marketable audience. Perhaps only because they arrive en-mass and end up slightly faceless. I’m sure it would be weird for most teachers to sell t-shirts in a class-room. It wasn’t particularly weird online, where we’re desensitised to being constantly sold things. And I may have only noticed this because I’m interested in how all these things fit together.

$ Ableton

The course was about learning Ableton Live. A commercial audio production tool. So at some point, the cost of Ableton had to be considered. Ableton offers a free 30 day trial, which works for this course and they kindly (or sensibly) agreed to let people taking the course start a new trial even if they’d used their 30 days already. Good manners like those are good for business. Anyway, I already owned Live 9 Intro (aka the cheap version), and for a three week intro course it does more than enough to learn the basics (I guess that’s why it’s called Intro?). But the course taught and encouraged the use of Live 9 Suite (the EUR599 rather than the EUR79 version). Until some people complained, the use of features in Suite was required to complete the final assignment. Reading between the lines, I doubt there was any deliberate commercial discussion around this planning, but the planning definitely didn’t stem from the question: ‘how can we keep the cost down for these beginners?’. At the end of the course there were discount codes to get 15% off purchasing anything from Ableton. I didn’t use Suite during the course, but I’m playing with it now on my own time and terms, and may end up spending money on it soon.

Reflections

It’s wonderful, but it’s not Wikipedia. The course opened a lot of doors, but mostly into places where I could spend money, which I am cautious about as a model for learning. It was valuable to me and prompted me to learn more about Ableton Live than I would have done in those three weeks without it. So I’m grateful for it. But I can’t in my heart think of this as a ‘shared public resource’.

For my own learning, I like deadlines. Preferably arbitrary. The fact that these Coursera courses are only available at certain times during the year, really works for me. But I struggle with the logic of this when I think about how best to provide learning material online to as many people as possible. The only MOOC style courses I have finished have been time-bound. I don’t know how many people this is true for though.

People will learn X to earn Y. For me this course was a form of hobby or entertainment, but much learning has a direct commercial interest for students as well as educators. Whether it’s for professional skills development, or building some perceived CV value.

There is no ‘free’ education, even if it says “free” on the homepage. There is always a cost, financial or otherwise. Sometimes the cost is borne by the educator, and sometimes the student. Both models have a place, but I get uncomfortable when one tries to look like the other. And if the world could only have one of these models for all of education I know which one I’d choose. Marketing fills enough of our daily content and claims enough brainprint as it is.

Conclusion

I thought I might find some conclusions in writing this, but that doesn’t always happen. There are a lot of interesting threads here.

So instead of a conclusion, you can have the song I submitted for my course assignment. It was fun to make. And I have this free-but-not-free course to thank for getting it done.

The week ahead 23 Feb 2015

First, I’ll note that even taking the time to write these short ‘note to self’ type blog posts each week takes time and is harder to do than I expected. Like so many priorities, the long term important things often battle with the short term urgent things. And that’s in a culture where working open is more than just acceptable, it’s encouraged.

Anyway, I have some time this morning sitting in an airport to write this, and I have some time on a plane to catch up on some other reading and writing that hasn’t made it to the top of the todo list for a few weeks. I may even get to post a blog post or two in the near future.

This week, I have face-to-face time with lots of colleagues in Toronto. Which means a combination of planning, meetings, running some training sessions, and working on tasks where timezone parity is helpful. It’s also the design team work week, and though I’m too far gone from design work to contribute anything pretty, I’m looking forward to seeing their work and getting glimpses of the future Webmaker product. Most importantly maybe, for a week like this, I expect unexpected opportunities to arise.

One of my objectives this week is working with Ops to decide where my time is best spent this year to have the most impact, and to set my goals for the year. That will get closer to a metrics strategy this year to improve on last years ‘reactive’ style of work.

IMG_0456If you’re following along for the exciting stories of my shed>to>office upgrades. I don’t have much to show today, but I’m building a new desk next and insulation is my new favourite thing. This photo shows the visible difference in heat loss after fitting my first square of insulation material to the roof.

Weeknotes: 30 Jan 2015

This works, but needs some work
This works, but needs some work

I don’t want these weeknotes to be a complete ‘done list’, as we have enough of those internally. This just a quick reflection on the extra objectives set for the week.

  • I moved outside into the MVP garden office (it’s still looking mostly like a shed).
    • This was thanks to the magical powers of powerline adapters which I only recently heard about. I still do not understand the sorcery that is transferring a high speed network through the existing electrical circuit, but it’s working without me needing to run any cabling.
  • I spent enough time chipping away at my processes and on ‘working open’ that I’m feeling good about it, and still enough time getting things done.
  • I cleaned out my ‘mofo-metrics’ Bugzilla backlog from 2014, and killed lots of tickets that weren’t relevant any more.
  • Setup my first Data Practices Review ticket as part of the new ‘Data Steward’ work I have acquired this year.

I feel like I’m getting in to the flow of 2015 a bit now.

Webmaking in the UK, and face-to-face events

One of this week’s conversations was with Nesta, about Webmaker usage within the UK and whether or not we have data to support the theory that face-t0-face events have an impact getting people involved in making on the web. These are two topics that interest me greatly.

I’m basically copying some of my notes into blog form so that the conversation isn’t confined to a few in-boxes.

And the TL;DR is our data represents what we’ve done, rather than any universal truth.

Our current data would support the hypothesis that face-to-face time is important for learning, but that would simply be because that’s how our program has been designed to date. In other words, our Webmaker tools were designed primarily for use in face-to-face events, which meant that adoption by ‘self-learners’ online is low because their is little guidance or motivation to play with our tools on your own. This year we’re making a stronger push on developing tools that can be used remotely, alongside our work on volunteer led face-to-face events. This will lead to a less biased overall data set in the future where we can begin to properly explore the impact on making and learning for people who do or don’t attend face-to-face events at various stages in their learning experience. In particular I’m keen to understand what factors help people transition from learners, to mentoring and supporting their peers.

I also took a quick look at the aggregate Google Analytics location data for the UK audience which I hadn’t done before and which re-enforces the point above.

Screen Shot 2015-01-30 at 11.14.29

Above: Traffic to Webmaker (loosely indicating an interest in the topic) is roughly distributed like a population map of the UK. This is what I expect to see of most location data.

Screen Shot 2015-01-30 at 11.17.25

Above: However, if you look at the locations of visitors who make something, there are lots of clusters around the UK and London is equaled by many other cities.

To-date, usage of the Webmaker tools has been driven by those who are using the tools to teach the web (i.e. Webmaker Mentors). But we also know there are large numbers of people who find Webmaker outside of the face-to-face event scenarios who need a better route into Webmaker’s offering.

The good news is that this year’s plans look after both sets of potential learners.

The week ahead: 26 Jan 2015

unrelatedphoto

I should have started the week by writing this, but I’ll do it quickly now anyway.

My current todo list.
List status: Pretty good. Mostly organized near the top. Less so further down. Fine for now.

Objectives to call out for this week.

  • Bugzilla and Github clean-out / triage
  • Move my home office out to the shed (depending on a few things)

+ some things that carry over from last week

  • Write a daily working process
  • Work out a plan for aligning metrics work with dev team heartbeats
  • Don’t let the immediate todo list get in the way of planning long term processes
  • Invest time in working open
  • Wrestle with multiple todo list systems until they (or I) work together nicely

The week ahead: 19 Jan 2015

January

If all goes to plan, I will:

  • Write a daily working process
  • Use a public todo list, and make it work
  • Catch up on more email from time off
  • Ship V1 of Webmaker Metrics retention dashboard
  • Work out a plan for aligning metrics work with dev team heartbeats
  • Don’t let the immediate todo list get in the way of planning long term processes
  • Invest time in working open
  • Wrestle with multiple todo list systems until they (or I) work together nicely
  • Survive a 5 day week (it’s been a while)
  • Write up final testing blog posts from EOY before those tests are forgotten
  • Book data practices kick-off meetings with all teams

To try and solve some of the process challenges, I’ve gone back to a tool I built a couple of years ago (Done by When) and I’m breaking it a little bit to make it useful to me again. This might end up being an evening time project to learn about some of the new tech the Webmaker team are using this year (particularly rewriting the front end with React). I find it useful to have a side-project to use as a playground for learning new things.

Anyway, have a great week. I’ll try and write up some more notes at the end.

From 2014 to 2015

I’ve had a couple of weeks off work and it’s been a good time to reflect on the year past, and the one ahead. And before I dive back into things on Friday morning, I wanted to get this post published. It’s a long one, and writing it was more for my benefit than yours ;)

On 2014

Cassie’s post on 2014 has claimed the perfect title already, but I’ll ditto that it was a hell of a year: New job, second baby, two house moves, one house purchase, one trip to Toronto, two to San Francisco, Mozfest in London, Mozlandia in Portland, finally finishing my degree and graduating, and continually adjusting our home-life around the amazing speed at which a two year old and a newborn can change in any 24 hour period.

Some reflections:

My job title might focus on Metrics, but I don’t just work with numbers, I work with people. And that makes me happy because people are the best. While I love finding ways to measure things, what’s infinitely more interesting is the process of connecting that measurement back to decisions other people make. I’m one year into this role and it’s turning out more like I imagined it from the job description than I thought would be possible, but perhaps that’s because I’ve been given the space to make it that way.

On that note, public speaking is still one of the best ways to learn something (though ‘public writing’ may be almost as useful). I’ve learned to not-hate giving presentations but I don’t rush into them either. I gave a talk in September on ‘Working with numbers and People / Human Beings’, which was a useful exercise in reflecting on my own work, and how I go about doing my day job (slides). It forced me to think about many things that will help me plan for 2015.

I’m in many teams, even though I’m in a team of one. I make a conscious effort to understand the culture, process and motivations of all the teams and people I work with, and I take it as a huge compliment when people forget I’m not just on their team, but rather I’m working with all the teams. This cross-team working is something I’ve gravitated to in my roles prior to Mozilla too, but Metrics is a topic that lends itself especially well to this kind of cross-team work.

Being on many teams can be exhausting. Especially if it’s a ‘planning workweek’, in say Portland, and you’re working with all the teams who are doing their 2015 planning and also working in the team who are specifically not planning, because it’s the End of Year fundraising campaign (which could more than fill the week’s working hours on its own). That looked a lot like working from 04:00 to 22:00 for a few days, though I reclaimed my downtime in party at the end of the week. That was a good night.

On that note, I exceeded my working capacity by the end of the year. There will always be more work than can be done, but it takes a while for a new role to settle, and for the organization to know what to ask of and expect from a new role like mine. And by the end of the year, I’d overstretched what I had the capacity to deliver. That’s normal, but I need to build working processes this year that expect more work requests than can be delivered rather than just saying yes to everything like I was able to do for most of 2014.

‘Janky’ solutions were the right thing in 2014 for most of my projects (and I absorbed the word Janky into my vocabulary). This year I built a lot of ‘temporary’ and ‘interim’ things, which were useful at the time and helped various people make decisions quickly but which weren’t designed for long term maintenance. For most of these projects that was the right decision, as so much strategy is changing in 2015. But moving forward, I want to build some more reusable infrastructure around this work.

I had an (almost) clean slate, technology wise. The role was new so I didn’t inherit a bunch of systems to maintain, but I also didn’t build up much infrastructure. As I think about technology solutions in 2015, I need to keep maintenance time for projects to a minimum, because it could quickly eat up my capacity to get new things done. This year I want to shift to longer term solutions though for some of the work that was ‘janky’ in 2014.

I learned (much more) JavaScript. Previously I’d dabbled in front-end JS leaning heavily on jQuery. But working on Webmaker.org projects, and building other apps and services around our contribution metrics projects I learned a fair amount about node.js development and also a chunk of D3, both of which I’ve come to like a lot. I also upped my git skills. So +1 for learning by doing.

I have loved and hated Bugzilla this year. It took me most of the year to get a Bugzilla based work management system in place and use this as my central record for all metrics work, but with a bunch of MoFo dev and planning work shifting to Github for issue management, my work and collaboration process is now in multiple systems. This is a headache for me, but I’ll find a way to make this work.

Working remotely is excellent, given enough face-to-face time. I’m fine with managing my own time at home, having completed my degree this way and having worked as a freelancer too, but keeping the work ‘real’ with video meetings is something I need. And while the face-to-face workweeks throughout the year are tough for taking me away from home and our young family, they are essential for building the relationships with the team. If we didn’t have the workweeks, the remote work would be much harder. And when it all adds up, I think I get more time with my kids than most working parents who never have to travel. This year, I was really happy with the balance.

Working across timezones is hard, sometimes. I love having the morning to tackle complex problems knowing I won’t get a phone call or email as everyone else is still fast asleep, but by the afternoon I hit a point where my calendar is quickly full of calls, and the people waking up are full of questions and I switch from ‘doing’ to ‘answering’ mode. This is usually fine, but it means my calls are back-to-back until 6pm my time when I instantly switch from a bunch of unresolved threads of work conversation in my head to step out of my office and play with son for an hour before he goes to bed. It’s amazing to get that time with my son at this age, but that context switch is really tough and I need a better process of recording those open threads while on the calls, so that I can relax until the following morning knowing things won’t be lost. When we hit busier periods across the org, I know I’ll lose more of my evenings to late video meetings, but I try to keep this in check with the rest of life’s needs.

I need to take more time off throughout the year. Otherwise it all piles up at the end of the year like it did this year (though the time off has been useful for settling in after our recent house move). Taking time off was hard in year one, when I was still finding my place, and working out what’s expected and required in a new role, but it’s important for long term productivity. It’s the time when you have room to breathe and think further ahead, and put the work into the context of personal life and the world we live in. I’m lucky to do work that I am personally interested in and committed to, but that can make switching off hard at times. When your work is online, it’s hard to avoid work without avoiding the internet.

I completed my degree this year, after a six-year-long year-off. I got myself a ‘BSc (Open)’. The ‘Open’ is because it was a ‘choose your own adventure’ type of qualification via the Open University. I’ve written about that story elsewhere, but the final qualification is a little bit of maths, a fair amount of computer science and finished off with some storytelling in a creative writing course. This was a combination which I thought was funny at the time when I registered, but now turns out to be what I do for a living.

In 2015

I want to learn more about learning. I am a compulsive learner but I know that doesn’t apply to everyone, and with Webmaker’s educational mission, I want to better understand how learning works. In part that connects to the metrics angle of my role via the scientific field of Learning Analytics, but also I just want to know more about this for myself.

I might try and write at least some code every day (how’s that for commitment?!). Though I’m not one for New Year’s resolutions, because I generally like to set myself new challenges all year round, on New Year’s Eve I recalled John Resig’s post about writing code every day and told my wife I would do this in 2015. Then on 2nd January I got a vomitting bug and took to bed for the best part of 3 days, failing my goal pretty quickly. Since then however, I have been hacking on Done by When in my evenings and really enjoying it. I’ll see how this goes. Overall I’d like to be coding more regularly, rather than in short intense bursts.

I’ll need to say no to some things. As noted above, I’ve reached capacity so need to think about what does and doesn’t get done throughout the year.

I need to manage myself as though I was a team. I can’t just attack whatever is on top of the todo list and keep letting the list grow. I will adopt some of the heartbeat principles used by Webmaker, and try to be strict with myself about allocating separate time for planning, doing, documenting, and communicating. It can’t all be doing. I’ll write more about my processes as I work them out.

I’ll leave this blog post at that, as it’s grown to quite a length. I’m back to work from tomorrow morning, and this afternoon I need to finishing making the spare room into a suitable working environment.

Catch you online soon.

Fundraising testing update

I wrote a post over on fundraising.mozilla.org about our latest round of optimization work for our End of Year Fundraising campaign.

We’ve been sprinting on this during the Mozilla all-hands workweek in Portland, which has been a lot of fun working face-to-face with the awesome team making this happen.

You can follow along with the campaign, and see how were doing at fundraising.mozilla.org

And of course, we’d be over the moon if you wanted to make a donation.

IMG_0373
These amazing people are working hard to build the web the world needs.