I'm Adam. Nice to meet you.
My Twitter bio:
Metrics Lead at Mozilla Foundation. Previously at WWF. Test, tinker, and question the answers.
I hesitantly post this, as I’m spending the evening looking at DALMOOC and hope to take part, but know I’m short on free time right now (what with a new baby and trying to buy a house) and starting the course late.
This is either the first in a series of blog posts about this course, or, we shall never talk about this again.
The course encourages open and distributed publishing of work and assessments, which makes answering this first ‘warm-up’ task feel like more of a commitment to the course than I can really make. But here goes…
Competency 0.1: Describe and navigate the distributed structure of DALMOOC, different ways to engage with peers and various technologies to manage and share personal learning.
DALMOOC offers and encourages learning experiences that span many online products from many providers but which all connect back to a core curriculum hosted on the edX platform. This ranges from learning to use 3rd party tools and software to interacting with peers on commercial social media platforms like Twitter and Facebook. Learners can pick the tools and engagement best suited to them, including an option to follow just the core curriculum within edX if they prefer to do so.
It actually feels a lot like how we work at Mozilla, which is overwhelming and disorientating at first but empowering in the long run.
Writing this publically, however lazily, has forced me to engage with the task much more actively than I might have just sitting back and watching a lecture and answering a quiz.
But I suspect that a fear of the web, and a lack of experience ‘working open’ would make this a terrifying experience for many people. The DALMOOC topic probably pre-selects for people with a higher than average disposition to work this way though, which helps.
If I find a moment, I’ll write about many of the fun and inspiring things I saw at Mozfest this weekend, but this post is about a single session I had the pleasure of hosting alongside Andrew, Doug and Simon; Learning Analytics for Good in the Age of Big Data.
We had an hour, no idea if anyone else would be interested, or what angle people would come to the session from. And given that, I think it worked out pretty well.
We had about 20 participants, and broke into four groups to talk about Learning Analytics from roughly 3 starting points (though all the discussions overlapped):
- Practical solutions to measuring learning as it happens online
- The ethical complications of tracking (even when you want to optimise for something positive – e.g. Learning)
- The research opportunities for publishing and connecting learning data
But, did anyone learn anything in our Learning Analytics session?
Well, I know for sure the answer is yes… as I personally learned things. But did anyone else?
I spoke to people later in the day who told me they learned things. Is that good enough?
As I watched the group during the session I saw conversations that bounced back and forth in a way that rarely happens without people learning something. But how does anyone else who wasn’t there know if our session had an impact?
How much did people learn?
This is essentially the challenge of Learning Analytics. And I did give this some thought before the session…
As a meta-exercise, everyone who attended the session had a question to answer at the start and end. We also gave them a place to write their email address and to link their ‘learning data’ to them in an identifiable way. It was a little bit silly, but it was something to think about.
This isn’t good science, but it tells a story. And I hope it was a useful cue for the people joining the session.
- We had about 20 participants
- 10 returned the survey (i.e. opted in to ‘tracking’), by answering question 1
- 5 of those answered question 2
- 5 gave their email address (not exactly the same 5 who answered both questions)
Here is our Learning Analytics data from our session
Is that demonstrable impact?
Even though this wasn’t a serious exercise. I think we can confidently argue that some people did learn, in much the same way certain newspapers can make a headline out of two data points…
What, and how much they learned, and if it will be useful later in their life is another matter.
Even with the deliberate choice of question which was almost impossible to not show improvement from start to end of the session, one respondent claims to be less sure what the session was about after attending (but let’s not dwell on that!).
Post-it notes and scribbles
If you were at the session, and want to jog your memory about what we talked about. I kind-of documented the various things we captured on paper.
I’m looking forward to exploring Learning Analytics in the context of Webmaker much more in 2015.
And to think that this was just one hour in a weekend full of the kinds of conversations that repeat in your mind all the way until next Mozfest. It’s exhausting in the best possible way.
I’m back at the screen after a week of paternity leave, and I’ll be working part-time for next two weeks while we settle in to the new family routine at home.
In the meantime, I wanted to mention a Mozilla contributor analysis project in case people would like to get involved.
We have a wiki page now, which means it’s a real thing. And here are some words my sleep-deprived brain prepared for you earlier today:
The goal and scope of the work:
Explore existing contribution datasets to look for possible insights and metrics that would be useful to monitor on an ongoing basis, before the co-incident workweek in Portland at the beginning of December.
- Stress-test our current capacity to use existing contribution data
- Look for actionable insights to support Mozilla-wide community building efforts
- Run ad-hoc analysis before building any ‘tools’
- If useful, prototype tools that can be re-used for ongoing insights into community health
- Build processes so that contributors can get involved in this metrics work
- Document gaps in our existing data / knowledge
- Document ideas for future analysis and exploration
I’m very excited that three members of the community have already offered to support the project and we’ve barely even started.
In the end, these numbers we’re looking at are about the community, and for the benefit of the community, so the more community involvement there is in this process, the better.
If you’re interested in data analysis, or know someone who is, send them the link.
This project is one of my priorities over the following 4-8 weeks. On that note, this looks quite appealing right now.
So I’m going make more tea and eat more biscuits.
- Removing the second sentence increases conversion rate (hypothesis = simplicity is good).
- The button text ‘Go!’ increased the conversion rate.
- Both variations on the headline increased conversion rate, but ‘Welcome to Webmaker’ performed the best.
- We should remove the bullet points on this landing page.
- The log-in option is useful on the page, even for a cold audience who we assume do not have accounts already.
- Repeating the ask ‘Sign-up for Webmaker’ at the end of the copy, even when it duplicates the heading immediately above, is useful. Even at the expense of making the copy longer.
- The button text ‘Create an account’ works better than ‘Sign up for Webmaker’ even when the headline and CTA in the copy are ‘Sign up for Webmaker’.
- These two headlines are equivalent. In the absence of other data we should keep the version which includes the brand name, as it adds one further ‘brand impression’ to the user journey.
- The existing blue background color is the best variant, given the rest of the page right now.
The Webmaker Testing Hub
If any of those “conclusions” sound interesting to you, you’ll probably want to read more about them on the Webmaker Testing Hub (it’s a fancy name for a list on a wiki).
This is where we’ll try and share the results of any test we run, and document the tests currently running.
And why that image for this blog post?
Because blog posts need and image, and this song came on as I was writing it. And I’m sure it’s a song about statistical significance, or counting, or something…
Here are a couple of notes about ‘Hack the snippet‘ that I wanted to make sure got documented.
- It significantly changed peoples’ predisposition to Webmaker before they arrived on the site
- Its ‘post-interaction’ click-through-rate was equivalent to most one-click snippets
Behind these observations, something special was happening in ‘Hack the snippet’. I can’t tell you exactly what it was that had the end-effect, but it’s worth remembering the effect.
1. It ‘warmed people up’ to Webmaker
- The ‘Hack the snippet’ snippet
- was shown to the same audience (Firefox users) as eight other snippet variations we ran during the campaign
- had the same % of users click through to the landing page
- had the same on-site experience on webmaker.org as all the other snippet variations we tested (the same landing page, sign-up ask etc)
- But when people who had interacted with ‘Hack the snippet’ landed on the website, they were more than three times as likely to signup for a webmaker account
Same audience, same engagement rate, same ask… but triple the conversion rate (most regular snippet traffic converted ~2%, ‘Hack the snippet’ traffic converted ~7%).
Something within that experience (and likely the overall quality of it) makes the Webmaker proposition more appealing to people who ‘hacked the snippet’. It could be one of many things: the simplicity, the guided learning, the feeling of power from editing the Firefox start page, the particular phrasing of the copy or many of the subtle design decisions. But whatever it was, it worked.
We need to keep looking for ways to recreate this.
Not everything we do going forwards needs to be a ‘Hack the snippet’ snippet (you can see how much time and effort went into that in the bug).
But when we think about these new-user experiences, we have a benchmark to compare things too. We know how much impact these things can have when all the parts align.
2. The ‘post-interaction’ CTR was as good as most one-click snippets
This is a quicker note:
- Despite the steps involved in completing the ‘Hack the snippet’ on page activity, the same total number of people clicked through when compared to a standard ‘one-click’ snippet.
- We got the same % of the audience to engage with a learning activity and then click through to the webmaker site, as we usually get just giving them a link directly to Webmaker
- This defies most “best practice” about minimizing number of clicks
Again, this doesn’t give us an immediate thing we can repeat, but it gives us a benchmark to build on.
This post is an attempt to capture some of the things we’ve learned from a few busy and exciting weeks working on the Webmaker new user funnel.
I will forget some things, there will be other stories to tell, and this will be biased towards my recurring message of “yay metrics”.
How did this happen?
As Dave pointed out in a recent email to Webmaker Dev list, “That’s a comma, not a decimal.”
What happened to increase new user sign-ups by 1,024% compared the previous month?
Is there one weird trick to…?
Sorry, I know you’d like an easy answer…
This growth is the result of a month of focused work and many many incremental improvements to the first-run experience for visitors arriving on webmaker.org from the promotion we’ve seen on the Firefox snippet. I’ll try to recount some of it here.
While the answer here isn’t easy, the good news is it’s repeatable.
While I get the fun job of talking about data and optimization (at least it’s fun when it’s good news), the work behind these numbers was a cross-team effort.
I think this model worked really well.
Where are these new Webmaker users coming from?
We can attribute ~60k of those new users directly to:
- Traffic coming from the snippet
- Who converted into users via our new Webmaker Landing pages
I’ve tried to go back over our meeting notes for the month and capture the variations on the funnel as we’ve iterated through them. This was tricky as things changed so fast.
This image below gives you an idea, but also hides many more detailed experiments within each of these pages.
With 8 snippets tested so far, 5 funnel variations and at least 5 content variables within each funnel we’ve iterated through over 200 variations of this new user flow in a month.
We’ve been able to do this and get results quickly because of the volume of traffic coming from the snippet, which is fantastic. And in some cases this volume of traffic meant we were learning new things quicker than we were able to ship our next iteration.
What’s the impact?
If we’d run with our first snippet design, and our first call to action we would have had about 1,000 new webmaker users from the snippet, instead of 60,000 (the remainder are from other channels and activities). Total new user accounts is up by ~1,000% but new users from the snippet specifically increased by around 6 times that.
One not-very-weird trick to growth hacking:
I said there wasn’t one weird trick, but I think the success of this work boils down to one piece of advice:
- Prioritize time and permission for testing, with a clear shared objective, and get just enough people together who can make the work happen.
It’s not weird, and it sounds obvious, but it’s a story that gets overlooked often because it doesn’t have the simple causation based hooked we humans look for in our answers.
It’s much more appealing when someone tells you something like “Orange buttons increase conversion rate”. We love the stories of simple tweaks that have remarkable impact, but really it’s always about process.
More Growth hacking tips:
- Learn to kill your darlings, and stay happy while doing it
- We worked overtime to ship things that got replaced within a week
- It can be hard to see that happen to your work when you’re invested in the product
- My personal approach is to invest my emotion in the impact of the thing being made rather than the thing itself
- But I had to lose a lot of A/B tests to realize that
- Your current page is your control
- Test ideas you think will beat it
- If you beat it, that new page is your new control
- Rinse and repeat
- Optimize with small changes (content polishing)
- Challenge with big changes (disruptive ideas)
- Focus on areas with the most scope for impact
- Use data to choose where to use data to make choices
- Don’t stretch yourself too thin
What happens next?
- We have some further snippet coverage for the next couple of weeks, but not at the same level we’ve had recently, so we’ll see this growth rate drop off
- We can start testing the funnel we’ve built for other sources of traffic to see how it performs
- We have infrastructure for spinning up and testing landing pages for many future asks
- This work is never done, but with any optimization you see declining returns on investment
- We need to keep reassessing the most effective place to spend our time
- We have a solid account sign-up flow now, but there’s a whole user journey to think about after that
- We need to gather up and share the results of the tests we ran within this process
Testing doesn’t have to be scary, but sometimes you want it to be.
TL;DR: Check out this graph!
Ever wondered how many Mozfest Volunteers also host events for Webmaker? Or how many code contributors have a Webmaker contributor badge? Now you can find out…
The reason the MoFo Contributor dashboard we’re working from at the moment is called our interim dashboard is because it’s combining numbers from multiple data sources, but the number of contributors is not de-duped across systems.
So if you’re counted as a contributor because you host an event for Webmaker, you will be double counted if you also file bugs in Bugzilla. And until now, we haven’t known what those overlaps look like.
This interim solution wasn’t perfect, but it’s given us something to work with while we’re building out Baloo and the cross-org areweamillionyet.org (and by ‘we’, the vast credit for Baloo is due to our hard working MoCo friends Pierros and Sheeri).
To help with prepping MoFo data for inclusion in Baloo, and by generally being awesome, JP wired up an integration database for our MoFo projects (skipping a night of sleep to ship V1!).
We’ve tweaked and tuned this in the last few weeks and we’re now extracting all sorts of useful insights we didn’t have before. For example, this integration database is behind quite a few of the stats in OpenMatt’s recent Webmaker update.
The downside to this is we will soon have a de-duped number for our dashboard, which will be smaller than the current number. Which will feel like a bit of a downer because we’ve been enthusiastically watching that number go up as we’ve built out contribution tracking systems throughout the year.
But, a smaller more accurate number is a good thing in the long run, and we will also gain new understanding about the multiple ways people contribute over time.
We will be able to see how people move around the project, and find that what looks like someone ‘stopping’ contributing, might be them switching focus to another team, for example. There are lots of exciting possibilities here.
And while I’m looking at this from a metrics point of view today, the same data allows us to make sure we say hello and thanks to any new contributors who joined this week, or to reach out and talk to long running active contributors who have recently stopped, and so on.
- Our MoFo dashboards now have trendlines based on known activity to date
- The recent uptick in activity is partly new contributors, and partly new recognition of existing contributors (all of which is good, but some of which is misleading for the trendline in the short term)
- Below is a rambling analogy for thinking about our contributor goals and how we answer the question ‘are we on track for 2014?’
- + if you haven’t seen it, OpenMatt has crisply summarized a tonne of the data and insights that we’ve unpicked during Maker Party
I was stacking logs over the weekend, and wondering if I had enough for winter, when it struck me that this might be a useful analogy for a post I was planning to write. So bear with me, I hope this works…
To be clear, this is an analogy about predicting and planning, not a metaphor for contributors*
So the trendline looks good, but…
Trendlines can be misleading.
What if our task was gathering and splitting logs?
We’re halfway through the year, and the log store is half full. The important questions is, ‘will it be full when the snow starts falling?‘
Well, it depends.
It depends how quickly we add new logs to the store, and it depends how many get used.
So let’s push this analogy a bit.
Before this year, we had scattered stacks of logs here and there, in teams and projects. Some we knew about, some we didn’t. Some we thought were big stacks of logs but were actually stacked on top of something else.
Setting a target was like building a log store and deciding to fill it. We built ours to hold 10,000 logs. There was a bit of guesswork in that.
It took a while to gather up our existing logs (build our databases and counting tools). But the good news is, we had more logs than we thought.
Now we need to start finding and splitting more logs*.
Switching from analogy to reality for a minute…
This week we added trendlines to our dashboard. These are two linear regression lines. One based on all activity for the year to-date, and one based on the most recent 4 weeks. It gives a quick feedback mechanism on whether recent actions are helping us towards to our targets and whether we’re improving over the year to-date.
These are interesting, but can be misleading given our current working practices. The trendline implies some form of destiny. You do a load of work recruiting new contributors, see the trendline is on target, and relax. But relaxing isn’t an option because of the way we’re currently recruiting contributors.
Switching back to the analogy…
We’re mostly splitting logs by hand.
Things happen because we go out and make them happen.
Hard work is the reason we have 1,800 Maker Party events on the map this year and we’re only half-way through the campaign.
There’s a lot to be said for this way of making things happen, and I think there’s enough time left in the year to fill the log store this way.
But this is not mathematical or automated, which makes trendlines based on this activity a bit misleading.
In this mode of working, the answer to ‘Are we on track for 2014?‘ is: ‘the log store will be filled… if we fill it‘.
Systems can be tested, tuned, modified and multiplied. In a world of ‘systems’ we can apply trendlines to our graphs that are much better predictors of future growth.
We should be experimenting with systems now (and we are a little bit). But we don’t yet know what the contributor growth system looks like that works as well as the analogous log splitting machines of the forestry industry. These are things to be invented, tested and iterated on, but I wouldn’t bet on them as the solution for 2014 as this could take a while to solve.
I should also state explicitly that systems are not necessarily software (or hardware). Technology is a relatively small part of the systems of movement building. For an interesting but time consuming distraction, this talk on Social Machines from last week’s Wikimania conference is worth a ponder:
Predicting 2014 today?
Even if you’re splitting logs by hand, you can schedule time to do it. Plan each month, check in on targets and spend more or less time as required to stay on track for the year.
This boils down to a planning exercise, with a little bit of guess work to get started.
In simple terms, you list all the things you plan to do this year that could recruit contributors, and how many contributors you think each will recruit. As you complete some of these activities you reflect on your predictions, and modify the plans and update estimates for the rest of the year.
Geoffrey has put together a training workshop for this, along with a spreadsheet structure to make this simple for teams to implement. It’s not scary, and it helps you get a grip on the future.
From there, we can start to feed our planned activity and forecast recruitment numbers into our dashboard as a trendline rather than relying solely on past activity.
The manual nature of the splitting-wood-like-activity means what we plan to do is a much more important predictor of the future than extrapolating what we have done in the past, and that changing the future is something you can go out and do.
*Contributors are not logs. Do not swing axes at them, and do not under any circumstances put them in your fireplace or wood burning stove.
We’re a little over halfway through the year now, and our dashboard is now good enough to tell us how we’re doing.
- The existing trend lines won’t get us to our 2014 goals
- but knowing this is helpful
- and getting there is possible
- Ask less: How do we count our contributors?
- Ask more: What are we doing to grow the contributor community? And, are we on track?
Changing the question
Our dashboard now needs to move from being a project to being a tool that helps us do better. After all, Mozilla’s unique strength is that we’re a community of contributors and this dashboard, and the 2014 contributor goal, exist to help us focus our workflows, decisions and investments in ways that empower the community. Not just for the fun of counting things.
The first half of the year focused us on the question “How do we count contributors?”. By and large, this has now been answered.
We need to switch our focus to:
- Are we on track?
- What are we doing to grow the contributor community?
Then repeating these two question regularly throughout the year, and adjusting our strategy as we go.
Are we on track?
Wearing my cold-dispassionate-metrics hat, and not my “I know how hard you’re all working already” hat, I have to say no (or, not yet).
I’m going to look at this team by team and then look at the All Mozilla Foundation view at the end.
Your task, for each graph below is to take an imaginary marker pen and draw the line for the rest of the year based on the data you can see to date. And only on the data you can see to-date.
- What does your trend line look like?
- Is it going to cross the dotted target line in 2014?
Based on the data to-date, I’d draw a flat line here. Although there are new contributors joining pretty regularly, the overall trend is flat. In marketing terms there is ‘churn'; not a nice term, but a useful one to talk about the data. To use other crass marketing terms, ‘retention’ is as important as ‘acquisition’ in changing the shape of this graph.
Dispassionately here, I’d have to draw a trend line that’s pointing slightly down. One thing to note in this view is that the Science Lab team have good historic data, so what we’re seeing here is the result of the size of the community in early 2013, and some drop-off from those people.
This graph is closest to what we want to see generally, i.e. pointing up. But I’ll caveat that with a couple of points. First, taking the imaginary marker pen, this isn’t going to cross the 2014 target line at the current rate. Second, unlike the Science Lab and OpenNews data above, much of this Appmaker counting is new. And when you count things for the first time, a 12 month rolling active total has a cumulative effect in the first year, which increases the appearance of growth, but might not be a long term trend. This is because Appmaker community churn won’t be a visible thing until next year when people could first drop out of the twelve month active time-frame.
This graph is the hardest to extend with our imaginary marker pen, especially with the positive incline we can see as Maker Party kicks off. The Webmaker plan expects much of the contributor community growth to come from the Maker Party campaign, so a steady incline was not the expectation across the year. But, we can still play with the imaginary marker pen.
I’d do the following exercise: In the first six months, active contributors grew by ~800 (~130 per month), so assuming that’s a general trend (big assumption) and you work back from 10k in December you would need to be at ~9,500 by the end of September. Mark a point at 9,500 contributors above the October tick and look at the angle of growth required throughout Maker Party to get there. That’s not impossible, but it’s a big challenge and I don’t have any historic data to make an informed call here.
Note: the Appmaker/Webmaker separation here is a legacy thing from the beginning of the year when we started this project. The de-duped datastore we’re working on next will allow us to graph: Webmaker Total > Webmaker Tools > Appmaker as separate graphs with separate goals, but which get de-duped and roll-up into the total numbers above, and in turn roll-up into the Mozilla wide total at areweamillionyet.org – this will better reflect the actual overlaps.
[ 0 contributors ]
The MoFo metrics team currently has zero active volunteer contributors, and based on the data available to date is trending absolutely flat. Action is required here, or this isn’t going to change. I also need to set a target. Growing 0 by 10X doesn’t really work. So I’ll aim for 10 volunteer contributors in 2014.
All Mozilla Foundation
Here we’re adding up the other graphs and also adding in ~900 people who contributed to MozFest in October 2013. That MozFest number isn’t counted towards a particular team and simply lifts the total for the year. There is no trend for the MozFest data because all the activity happened at once, but if there wasn’t a MozFest this year (don’t worry, there is!) in October the total line would drop by 900 in a single week. Beyond that, the shape of this line is the cumulative result of the team graphs above.
In Q3, we’ll be able to de-dupe this combined number as there are certainly contributors working across MoFo teams. In a good way, our total will be less that the sum of our parts.
Where do we go from here?
First, don’t panic. Influencing these trend lines is not like trying to shift a nation’s voting trends in the next election. Much of this is directly under our control, or if not ‘control’, then it’s something we can strongly influence. So long as we work on it.
Next, it’s important to note that this is the first time we’ve been able to see these trends, and the first time we can measure the impact of decisions we make around community building. Growing a community beyond a certain scale is not a passive thing. I’ve found David Boswell’s use of the term ‘intentional’ community building really helpful here. And much more tasteful than my marketing vocabulary!
These graphs show where we’re heading based on what we’re currently doing, and until now we didn’t know if we were doing well, or even improving at all. We didn’t have any feedback mechanism on decisions we’d make relating to community growth. Now we do.
Here are some initial steps that can help with the ‘measuring’ part of this community building task.
Going back to the marker pen exercise, take another imaginary color and rather than extrapolate the current trend, draw a positive line that gets you to your target by the end of the year. This doesn’t have to be a straight line; allow your planned activity to shape the growth you want to see. Then ask:
- Where do you need to be in Aug, Sep, Oct, Nov, Dec?
- How are you going to reach each of these smaller steps?
Schedule a regular check-in that focuses on growing your contributor community and check your dashboard:
- Are your current actions getting you to your goals?
- What are the next actions you’re going to take?
The first rule of fundraising is ‘Ask for money’. People often overlook this. By the same measure, are you asking for contributions?
- How many people are you asking this week or month to get involved?
- What percentage of them do you expect to say yes and do something?
Multiply those numbers together and see if it that prediction can get you to your next step towards your goal.
Asking these questions alone won’t get us to our goals, but it helps us to know if our current approach has the capacity to get there. If it doesn’t we need to adjust the approach.
Those are just the numbers
I could probably wrap up this check-in from a metrics point of view here, but this is not a numbers game. The Total Active Contributor number is a tool to help us understand scale beyond the face-to-face relationships we can store in our personal memories.
We’re lucky at Mozilla that so many people already care about the mission and want to get involved, but sitting and waiting for contributors to show up is not going to get us to our goals in 2014. Community building is an intentional act.
- I need to work on this as much as anyone else. I’ll be working with the CBT Team to learn from their experience and build a Metrics contribution pathway. https://bugzilla.mozilla.org/show_bug.cgi?id=1033359
- I recommend this great read from the addons.mozilla.org reviewer team who have 10+ years of experience we can learn from: http://mozamy.wordpress.com/2014/07/14/amo-reviewers/
- And, this wiki page leads to a wealth of Mozilla knowledge and shared resources around community building. https://wiki.mozilla.org/Contribute
Here’s to setting new trends.