The week ahead 23 Feb 2015

First, I’ll note that even taking the time to write these short ‘note to self’ type blog posts each week takes time and is harder to do than I expected. Like so many priorities, the long term important things often battle with the short term urgent things. And that’s in a culture where working open is more than just acceptable, it’s encouraged.

Anyway, I have some time this morning sitting in an airport to write this, and I have some time on a plane to catch up on some other reading and writing that hasn’t made it to the top of the todo list for a few weeks. I may even get to post a blog post or two in the near future.

This week, I have face-to-face time with lots of colleagues in Toronto. Which means a combination of planning, meetings, running some training sessions, and working on tasks where timezone parity is helpful. It’s also the design team work week, and though I’m too far gone from design work to contribute anything pretty, I’m looking forward to seeing their work and getting glimpses of the future Webmaker product. Most importantly maybe, for a week like this, I expect unexpected opportunities to arise.

One of my objectives this week is working with Ops to decide where my time is best spent this year to have the most impact, and to set my goals for the year. That will get closer to a metrics strategy this year to improve on last years ‘reactive’ style of work.

IMG_0456If you’re following along for the exciting stories of my shed>to>office upgrades. I don’t have much to show today, but I’m building a new desk next and insulation is my new favourite thing. This photo shows the visible difference in heat loss after fitting my first square of insulation material to the roof.

The week ahead: 26 Jan 2015

unrelatedphoto

I should have started the week by writing this, but I’ll do it quickly now anyway.

My current todo list.
List status: Pretty good. Mostly organized near the top. Less so further down. Fine for now.

Objectives to call out for this week.

  • Bugzilla and Github clean-out / triage
  • Move my home office out to the shed (depending on a few things)

+ some things that carry over from last week

  • Write a daily working process
  • Work out a plan for aligning metrics work with dev team heartbeats
  • Don’t let the immediate todo list get in the way of planning long term processes
  • Invest time in working open
  • Wrestle with multiple todo list systems until they (or I) work together nicely

#DALMOOC structure

I hesitantly post this, as I’m spending the evening looking at DALMOOC and hope to take part, but know I’m short on free time right now (what with a new baby and trying to buy a house) and starting the course late.

This is either the first in a series of blog posts about this course, or, we shall never talk about this again.

The course encourages open and distributed publishing of work and assessments, which makes answering this first ‘warm-up’ task feel like more of a commitment to the course than I can really make. But here goes…

Competency 0.1: Describe and navigate the distributed structure of DALMOOC, different ways to engage with peers and various technologies to manage and share personal learning.

DALMOOC offers and encourages learning experiences that span many online products from many providers but which all connect back to a core curriculum hosted on the edX platform. This ranges from learning to use 3rd party tools and software to interacting with peers on commercial social media platforms like Twitter and Facebook. Learners can pick the tools and engagement best suited to them, including an option to follow just the core curriculum within edX if they prefer to do so.

It actually feels a lot like how we work at Mozilla, which is overwhelming and disorientating at first but empowering in the long run.

Writing this publically, however lazily, has forced me to engage with the task much more actively than I might have just sitting back and watching a lecture and answering a quiz.

But I suspect that a fear of the web, and a lack of experience ‘working open’ would make this a terrifying experience for many people. The DALMOOC topic probably pre-selects for people with a higher than average disposition to work this way though, which helps.

Something special within ‘Hack the snippet’

Here are a couple of notes about ‘Hack the snippet‘ that I wanted to make sure got documented.

  1. It significantly changed peoples’ predisposition to Webmaker before they arrived on the site
  2. Its ‘post-interaction’ click-through-rate was equivalent to most one-click snippets

Behind these observations, something special was happening in ‘Hack the snippet’. I can’t tell you exactly what it was that had the end-effect, but it’s worth remembering the effect.

1. It ‘warmed people up’ to Webmaker

  • The ‘Hack the snippet’ snippet
    • was shown to the same audience (Firefox users) as eight other snippet variations we ran during the campaign
    • had the same % of users click through to the landing page
    • had the same on-site experience on webmaker.org as all the other snippet variations we tested (the same landing page, sign-up ask etc)
  • But when people who had interacted with ‘Hack the snippet’ landed on the website, they were more than three times as likely to signup for a webmaker account

Same audience, same engagement rate, same ask… but triple the conversion rate (most regular snippet traffic converted ~2%, ‘Hack the snippet’ traffic converted ~7%).

Something within that experience (and likely the overall quality of it) makes the Webmaker proposition more appealing to people who ‘hacked the snippet’. It could be one of many things: the simplicity, the guided learning, the feeling of power from editing the Firefox start page, the particular phrasing of the copy or many of the subtle design decisions. But whatever it was, it worked.

We need to keep looking for ways to recreate this.

Not everything we do going forwards needs to be a ‘Hack the snippet’ snippet (you can see how much time and effort went into that in the bug).

But when we think about these new-user experiences, we have a benchmark to compare things too. We know how much impact these things can have when all the parts align.

2. The ‘post-interaction’ CTR was as good as most one-click snippets

This is a quicker note:

  • Despite the steps involved in completing the ‘Hack the snippet’ on page activity, the same total number of people clicked through when compared to a standard ‘one-click’ snippet.
  • We got the same % of the audience to engage with a learning activity and then click through to the webmaker site, as we usually get just giving them a link directly to Webmaker
    • This defies most “best practice” about minimizing number of clicks

Again, this doesn’t give us an immediate thing we can repeat, but it gives us a benchmark to build on.

Contributors counting… contributors?

We now have a reasonably organized Mozilla Foundation Metrics Wiki Hub Page Thing.

While my priority to date this year has been working out how MoFo teams count their contributors, I thought I should also take the time to open up this metrics work in a way that contributors can get involved, if that’s what takes their fancy. After all, contributor metrics are only as good as the systems they help us improve, and in turn the contributors they help us empower. :)

As with many good things in the world of open source, this includes a mailing list.

So here’s by blurb if you’d consider signing up:

The mofo-metrics mailing list:

“An open community mailing list for volunteers and staff interested in Mozilla Foundation Metrics. What are the numbers, graphs and other data points that can help the Mozilla Foundation to better promote openness, innovation and participation on the Internet? Sign up and help us answer that question.”

I’m not 100% sure what contribution will look like in metrics-land, but I’m happy find out and to try and make this work.

Are we on track with our 2014 contributor goals?

I presented a version of this on the Mozilla Foundation staff call yesterday, and thought it’s worth a write-up for those who weren’t on the call and those in MoCo working on related things.

Some Context:

One of the cross Mozilla goals this year is to “10X” the number of active contributors, with a longer term goal of growing to a million Mozillians.

When the 10X goal was set we weren’t really sure what X was, for valid reasons; defining contribution is as much art as it is science, and the work plans for this year include building the tools to measure this in a systematic way. The goals justify the tools, and vice versa. Chicken and egg.

2,000 contributors were invited to the summit, so the target was set at 20k active contributors shared between MoCo and MoFo. MoFo have been working to a target of 10k contributors but in practice this isn’t going to be a clean 50/50 split and there will be overlap in contributors across teams, projects and MoFo/MoCo. For example, 10k MoCo contributors + 10k MoFo contributors could = 19k Mozilla contributors.

When I joined in January, each of the MoFo teams did some (slightly tedious) manual counting and estimated their contributor numbers for 2013, and we added these up to a theoretical 5,600 contributors. This was our baseline number. Useful to an order or magnitude, but not precise.

This 5,600 number suggests that 10k contributors was quite far off 10X contributors based on these January estimates, but 10k is still going to be a challenging goal. At 10X we’d have been aiming for 50k+ contributors.

From the data that’s emerging, 10k active contributors to MoFo feels like a sane but stretching target.

With the recent forming of the Badge Alliance, some MoFo goals are now Badge Alliance goals, and the same goes for counting people contributing to parts of the Open Badge ecosystem. As a result, our theoretical 5,600 contributor number got smaller. It’s now 4,200.

So 4,200 is where we assumed we started this year, but we haven’t proved this yet. And realizing this measurement has been our priority metrics project this year.

How are we doing so far?

We’ve been automating ways to count these ‘theoretical’ contributors, and feeding them into our dashboard.

But to-date as we’ve looked at the dashboard, and the provable number was 1,000 or 2,000 or so, we would then say “but the real number is actually closer to 5,000″. Which means the dashboard hasn’t been very useful yet, as the theoretical number always trumped the provable but incomplete number.

This will change in the next few weeks.

We’re now nearly counting ‘live’, all of those theoretical pots of contribution.

And the dashboard is at 2,800.

Once we add the Webmaker mentors who complete their training this year, and anything else that goes into the ad-hoc contribution logger, we’re basically at our real comparison point to that theoretical number, and we can drop the ‘theoretical’ bit.

If there’s a thousand mentors and another four hundred contributors added to the ad-hoc logger, our theoretical estimate will be remarkably close to reality. Except, that it’s six months behind where we thought it would be.

We’re getting close to that 4,200, but we expected (albeit pretty loosely) to be there in January.

This either means that:

(A) the growth shown on the graph to-date is an artifact of missing historical data, and we’re actually trending pretty flat.

(B) our 2013 estimates were too high and we started this year with fewer contributors than we thought, but we’ve been growing to date.

As we don’t have time-stamped historical data for some of these things, we’re not going to know which for sure. But either way, we now need to increase the rate at which we on-board new contributors to hit 10k by the end of the year.

There are plans in place for growing contribution numbers, but this is going to be active work.

Whether that’s converting new webmaker users who join us through Maker Party, or reducing barriers to contributing code or, actively  going out and asking people if they want to contribute. Growing that contributor number is going to be a combination of good processes and marketing.

Also to note

I’ll be making this MoFo total number smaller by X% when we integrate the data into a single location and de-dupe people across these  activities. But we don’t know what X% is yet. That’s just something to be aware of.

In relation to the points on there not being a clear MoCo/MoFo split in where people contribute, we’re much more directly connecting up the systems and processes now. We’ll have more to share on this in the coming weeks.

Tracking the status of the dashboard

Contributor Dashboard Status Update (‘busy work’?)

While I’m always itching to get on with doing the work that needs doing, I’ve spent this morning writing about it instead. Part of me hates this, but another realizes this is valuable. Especially when you’re working remotely and the project status in your head is of no use to your colleagues scattered around the globe.

So here’s the updated status page on our Mozilla Foundation Contributor Dashboard, and some progress on my ‘working open‘.

Filing bugs, linking them to each other, and editing wiki pages can be tedious work (especially wiki tables that link to bugs!) but the end result is very helpful, for me as well as those following and contributing to the project.

And a hat-tip to Pierros, whose hard-work on the project Baloo wiki page directly inspired the formatting here.

Now, back to doing! :)

Back in the open – priorities this week

The arrow is sort-of pointing to the Mozilla SF office
The arrow is sort-of pointing to the Mozilla SF office

I’m returning to the regular tasks today after a couple of weeks out of routine; one week at Mozilla Foundation’s ‘All-hands’ meeting, and one week away from the screen spending quality time with my family.

I will try and get back to the regular blogging that that I was doing at the beginning of the year, and will be working on ‘working open’ this week in particular. Even if it’s shorter posts like this one.

Priorities as I get started today:

  • Wiring up the last data sources for our interim contributor dashboard solution
  • Working with Mozilla Corporation counterparts to see what we can share and re-purpose from our interim dashboard for wider Mozilla use
    • Starting with opening up the ad-hoc logger for MoCo use
  • Getting some key tracking in place ahead of the Maker Party campaign for a conversion rate goal we set at all hands
  • Working on a test install of Metrics Grimoire for MoFo, that could be expanded to Mozilla in general
  • Moving much more of my workflow into Bugzilla (from 10% to something more like 90%)

Plus replying to a couple of weeks worth of emails, and the tasks lurking within them ;)

Here’s to a productive week.

 

What I see in these graphs of Github contribution

Context: Last week I shared a few graphs (1, 2, 3, 4) looking at data from our repositories on Github, extracted using this Gitribution app thing, as part of our work to dashboard contributor numbers for the Mozilla Foundation.

I didn’t comment on the graphs at the time because I wanted time for others to look at them without my opinions skewing what they might see. This follow up post is a walk-through of some things I see in the graphs/data.

The real value in looking at data is finding ways to make things better by challenging ourselves, and being honest about what the numbers show, so this will be as much about questions as answers…

Also, publishing this last week flagged up some missing repositories and identified some other members of staff so these graphs are based on the latest version of the data (there was no impact on shapes, but some numbers will be different).

What time of day do people contribute (UTC)?

By Hour of DayOur paid staff who are committing code are mostly in US/Canadian timezones and it make sense that most of their commits are during these hours (graphed by UTC). But, what caught my attention here is that the volunteer contribution times follow the same shape.

Questions to ask:

  • Do volunteer contributions follow the same shape because contributing code has a dependency on being able to talk in real time with staff? For example in IRC. If so, is this a bottleneck for contributing code?
  • If not, what is creating this shape for volunteer contributors? Perhaps it’s biased to timezones where more people are interested in the things we are building, and potentially biased by language? But looking at support for Maker Party and other activities there is a global audience for our tools.
  • What does a code contribution pathway look like for people in the 0300-1300UTC times? Is there anything we can do to make things easier or more appealing?

The shape of volunteer contributions

ShapeThe shape of this graph is pretty typical for any kind of volunteering or activity involving human interactions. It’s close to a power law graph with a long-tail.

If you’ve not looked at a data set like this before, don’t panic that so many people only make a single contribution. At the same time, don’t use the knowledge that this is typical not to ask questions about how we can be better.

Lots of people want to get involved in volunteering projects but often their good intentions don’t align with their actual available free time. I say this as someone who signs up for more things than fit into my available hours for personal projects.

The two questions I want to ask of this graph are:

  1. Where could our efforts to support contributors best influence the overall shape?
  2. What does this look like at 10 x scale?

So, starting with where we could influence shape… my opinion (no data here) says to think about people in this range.Shape HighlightTo the left of this highlighted area people are already making code contributions over and above even many staff. Shower them in endless gratitude! But I don’t think they don’t need practical help from us.  To the right of this highlighted area is the natural long tail. Supporting that bigger group of people for single-touch interactions is about clear documentation and easy to follow processes. But I think the group of people roughly highlighted in that graph are people we can reach out to. These people potentially have capacity to do more. We should find out what they are interested in, what they want to get out of contribution and build relationships with them. In practical terms, we have finite time to invest in direct relationships with contributors. I think this is an effective place to invest some of that time.

I think the second question is more  challenging. What does this look like at 10 x scale?

In 2013, ~50 people made a one-time contribution.

  • What do we need in place for 500 people to make a one-time code contribution?
  • Do we have 500 suitable ‘first’ bugs for 2014?
  • Is the amount of setup work required to contribute to our tools appropriate for people making a single contribution?
  • If not, is that a blocker to growing contributor numbers?

In 2013, there were ~1,500 code commits by volunteers.

  • What do we need in place for 15,000 activities on top of planned staff activity?
  • How does this much activity align towards a common product roadmap?
  • How is it scheduled, allocated, reviewed and shipped?

When planning to work with 10 x contributor numbers, possibly the biggest shift to consider is the ratio of staff to volunteers:

ContributorRatio

  • How does impact on time allocated for code reviews?
  • How do we write bugs?
  • How we prioritize bugs? Etc.
  • Even, what does an IRC channel or a dev maling list look like after this change?

Other questions to ask:

  • What do we think is the current ‘ceiling’ on our contributor numbers for people writing code?
    • Is it the number of developers who know about our tools and want to help? (i.e. a ‘marketing’ challenge to inspire more people)
    • Is it the amount of suitable work ready and available for people who want to help? (are we losing people who want to help because it’s too hard to get involved?)
    • Both? With any bias?

 What do you think?

I’m only one set of eyes on this, so please challenge my observations and feel free to build on this too.

Also, as the data in here is publicly accessible already I think I can publish this Tableau view as an interactive tool you can play with, but I need to check the terms first.

The Language of Contribution Metrics

TL;DR: Share your thoughts on the language we use around contribution metrics here (anyone can contribute): https://etherpad.mozilla.org/contributors-dashboard-language

Then if you have the time, here are some of my thoughts on this topic…

What does language have to do with metrics?

You’d be forgiven for thinking that working with data and metrics is a clean and scientific-like process of running queries against a database or two and generating a report. In many ways, I’m glad it’s not as simple as that.

Metrics are only as good as the things they enable us to improve. Which means while metrics need to be grounded in good clean data, they are primarily for people; and not just for people to read.

In their best incarnation, metrics motivate people to change things for the better.

At this scale, motivating people is definitely more art than science which gets us to the topic of this post: the language that frames our metrics.

The project this conversation relates to is building a dashboard to measure the number of active contributors to the Mozilla Foundation. Counting is a *reasonably* clean task on it’s own, but the reason this project exists is to support our goals to grow Mozilla at scale (10x at first, 1 Million Mozillians in time).

The language we use to frame the numbers on this dashboard does impact on how well the dashboard motivates us to ask tough questions of our plans and processes in relation to this goal. So it’s an important part of the dashboard UX.

My intro to this project was framed as such: “Our goal for 2014 is to ship 10,000 contributors”. And as someone who likes agile development, hacking things together and getting things done, ‘ship’ is a word that appeals to me and I think resonates for many at Mozilla, but it’s also internal parlance. Not secret by any means, but intended for a particular audience.

Where this language becomes ‘trickyish’, is having this conversation in the open (our plans are open and acknowledge this challenge). Our contributors are not a product, and the word ‘shipped’ might not sit right with them.

So how do we talk about growing contribution without risking taking away an individual’s feeling that contribution is something they choose to do?

It’s not a challenge unique to Mozilla, but our approach to working open might enable a unique solution.

This anecdote is from my time at WWF, and I think highlights the risk we are talking about:

I remember looking at supporter comments in response to the question “what prompted you to give?” which we asked at the end of our donation process. Though this story accounts for the minority of our respondents, there was a recurring theme where people’s records show they have responded to something like a unique marketing URL in a TV ad and donated to the particular issue highlighted by the ad in question; but they would go on to make comments like this: “Nothing *prompted* me to donate. I did so off my own back, because I care about [tigers/pandas/etc] and I myself decided I should do something to help.”

Many people (supporters/volunteers) strongly want to think they are entirely responsible for the things they choose to do.

But organisations recruiting supporters at scale know that you have to actively do things to bring people on board – from marketing and advertising through to welcoming and community management and much more. Fundraising 101: if you don’t ask you don’t get.

Traditional non-profit organisations, due to the economic pressures of effectively fundraising, skip this conversation and focus on the story of how support directly impacts the end goals of their mission… e.g. £10 can X. They cannot afford to lose the profitability of their fundraising in order to better talk about the challenge and process of fundraising (at least at scale) even though these business-like processes exist to support the same underlying mission that supporters care about.

But Mozilla’s way of working is far from traditional, so I think we’re in a great place to talk about why and how we’re counting contribution and more excitingly we can open up the tools that we hope can motivate staff to grow contributor numbers, so they can be used by the community too (this is another blog post to follow).

For now though, let’s talk about the word ‘shipping’.

Does the phrase ‘shipping contributors’ help you think at scale? Or does it sit uneasy? What word would you use instead?

Please join the conversation here:
https://etherpad.mozilla.org/contributors-dashboard-language