Weeknotes: 23 Jan 2015

unrelated photo

unrelated photo

I managed to complete roughly five of my eleven goals for the week.

  • Made progress on (but have not cracked) daily task management for the newly evolving systems
  • Caught up on some email from time off, but still a chunk left to work through
  • Spent more time writing code than expected
  • Illness this week slowed me down
  • These aren’t very good weeknotes, but perhaps better than none.

 

The week ahead: 19 Jan 2015

January

If all goes to plan, I will:

  • Write a daily working process
  • Use a public todo list, and make it work
  • Catch up on more email from time off
  • Ship V1 of Webmaker Metrics retention dashboard
  • Work out a plan for aligning metrics work with dev team heartbeats
  • Don’t let the immediate todo list get in the way of planning long term processes
  • Invest time in working open
  • Wrestle with multiple todo list systems until they (or I) work together nicely
  • Survive a 5 day week (it’s been a while)
  • Write up final testing blog posts from EOY before those tests are forgotten
  • Book data practices kick-off meetings with all teams

To try and solve some of the process challenges, I’ve gone back to a tool I built a couple of years ago (Done by When) and I’m breaking it a little bit to make it useful to me again. This might end up being an evening time project to learn about some of the new tech the Webmaker team are using this year (particularly rewriting the front end with React). I find it useful to have a side-project to use as a playground for learning new things.

Anyway, have a great week. I’ll try and write up some more notes at the end.

From 2014 to 2015

I’ve had a couple of weeks off work and it’s been a good time to reflect on the year past, and the one ahead. And before I dive back into things on Friday morning, I wanted to get this post published. It’s a long one, and writing it was more for my benefit than yours ;)

On 2014

Cassie’s post on 2014 has claimed the perfect title already, but I’ll ditto that it was a hell of a year: New job, second baby, two house moves, one house purchase, one trip to Toronto, two to San Francisco, Mozfest in London, Mozlandia in Portland, finally finishing my degree and graduating, and continually adjusting our home-life around the amazing speed at which a two year old and a newborn can change in any 24 hour period.

Some reflections:

My job title might focus on Metrics, but I don’t just work with numbers, I work with people. And that makes me happy because people are the best. While I love finding ways to measure things, what’s infinitely more interesting is the process of connecting that measurement back to decisions other people make. I’m one year into this role and it’s turning out more like I imagined it from the job description than I thought would be possible, but perhaps that’s because I’ve been given the space to make it that way.

On that note, public speaking is still one of the best ways to learn something (though ‘public writing’ may be almost as useful). I’ve learned to not-hate giving presentations but I don’t rush into them either. I gave a talk in September on ‘Working with numbers and People / Human Beings’, which was a useful exercise in reflecting on my own work, and how I go about doing my day job (slides). It forced me to think about many things that will help me plan for 2015.

I’m in many teams, even though I’m in a team of one. I make a conscious effort to understand the culture, process and motivations of all the teams and people I work with, and I take it as a huge compliment when people forget I’m not just on their team, but rather I’m working with all the teams. This cross-team working is something I’ve gravitated to in my roles prior to Mozilla too, but Metrics is a topic that lends itself especially well to this kind of cross-team work.

Being on many teams can be exhausting. Especially if it’s a ‘planning workweek’, in say Portland, and you’re working with all the teams who are doing their 2015 planning and also working in the team who are specifically not planning, because it’s the End of Year fundraising campaign (which could more than fill the week’s working hours on its own). That looked a lot like working from 04:00 to 22:00 for a few days, though I reclaimed my downtime in party at the end of the week. That was a good night.

On that note, I exceeded my working capacity by the end of the year. There will always be more work than can be done, but it takes a while for a new role to settle, and for the organization to know what to ask of and expect from a new role like mine. And by the end of the year, I’d overstretched what I had the capacity to deliver. That’s normal, but I need to build working processes this year that expect more work requests than can be delivered rather than just saying yes to everything like I was able to do for most of 2014.

‘Janky’ solutions were the right thing in 2014 for most of my projects (and I absorbed the word Janky into my vocabulary). This year I built a lot of ‘temporary’ and ‘interim’ things, which were useful at the time and helped various people make decisions quickly but which weren’t designed for long term maintenance. For most of these projects that was the right decision, as so much strategy is changing in 2015. But moving forward, I want to build some more reusable infrastructure around this work.

I had an (almost) clean slate, technology wise. The role was new so I didn’t inherit a bunch of systems to maintain, but I also didn’t build up much infrastructure. As I think about technology solutions in 2015, I need to keep maintenance time for projects to a minimum, because it could quickly eat up my capacity to get new things done. This year I want to shift to longer term solutions though for some of the work that was ‘janky’ in 2014.

I learned (much more) JavaScript. Previously I’d dabbled in front-end JS leaning heavily on jQuery. But working on Webmaker.org projects, and building other apps and services around our contribution metrics projects I learned a fair amount about node.js development and also a chunk of D3, both of which I’ve come to like a lot. I also upped my git skills. So +1 for learning by doing.

I have loved and hated Bugzilla this year. It took me most of the year to get a Bugzilla based work management system in place and use this as my central record for all metrics work, but with a bunch of MoFo dev and planning work shifting to Github for issue management, my work and collaboration process is now in multiple systems. This is a headache for me, but I’ll find a way to make this work.

Working remotely is excellent, given enough face-to-face time. I’m fine with managing my own time at home, having completed my degree this way and having worked as a freelancer too, but keeping the work ‘real’ with video meetings is something I need. And while the face-to-face workweeks throughout the year are tough for taking me away from home and our young family, they are essential for building the relationships with the team. If we didn’t have the workweeks, the remote work would be much harder. And when it all adds up, I think I get more time with my kids than most working parents who never have to travel. This year, I was really happy with the balance.

Working across timezones is hard, sometimes. I love having the morning to tackle complex problems knowing I won’t get a phone call or email as everyone else is still fast asleep, but by the afternoon I hit a point where my calendar is quickly full of calls, and the people waking up are full of questions and I switch from ‘doing’ to ‘answering’ mode. This is usually fine, but it means my calls are back-to-back until 6pm my time when I instantly switch from a bunch of unresolved threads of work conversation in my head to step out of my office and play with son for an hour before he goes to bed. It’s amazing to get that time with my son at this age, but that context switch is really tough and I need a better process of recording those open threads while on the calls, so that I can relax until the following morning knowing things won’t be lost. When we hit busier periods across the org, I know I’ll lose more of my evenings to late video meetings, but I try to keep this in check with the rest of life’s needs.

I need to take more time off throughout the year. Otherwise it all piles up at the end of the year like it did this year (though the time off has been useful for settling in after our recent house move). Taking time off was hard in year one, when I was still finding my place, and working out what’s expected and required in a new role, but it’s important for long term productivity. It’s the time when you have room to breathe and think further ahead, and put the work into the context of personal life and the world we live in. I’m lucky to do work that I am personally interested in and committed to, but that can make switching off hard at times. When your work is online, it’s hard to avoid work without avoiding the internet.

I completed my degree this year, after a six-year-long year-off. I got myself a ‘BSc (Open)’. The ‘Open’ is because it was a ‘choose your own adventure’ type of qualification via the Open University. I’ve written about that story elsewhere, but the final qualification is a little bit of maths, a fair amount of computer science and finished off with some storytelling in a creative writing course. This was a combination which I thought was funny at the time when I registered, but now turns out to be what I do for a living.

In 2015

I want to learn more about learning. I am a compulsive learner but I know that doesn’t apply to everyone, and with Webmaker’s educational mission, I want to better understand how learning works. In part that connects to the metrics angle of my role via the scientific field of Learning Analytics, but also I just want to know more about this for myself.

I might try and write at least some code every day (how’s that for commitment?!). Though I’m not one for New Year’s resolutions, because I generally like to set myself new challenges all year round, on New Year’s Eve I recalled John Resig’s post about writing code every day and told my wife I would do this in 2015. Then on 2nd January I got a vomitting bug and took to bed for the best part of 3 days, failing my goal pretty quickly. Since then however, I have been hacking on Done by When in my evenings and really enjoying it. I’ll see how this goes. Overall I’d like to be coding more regularly, rather than in short intense bursts.

I’ll need to say no to some things. As noted above, I’ve reached capacity so need to think about what does and doesn’t get done throughout the year.

I need to manage myself as though I was a team. I can’t just attack whatever is on top of the todo list and keep letting the list grow. I will adopt some of the heartbeat principles used by Webmaker, and try to be strict with myself about allocating separate time for planning, doing, documenting, and communicating. It can’t all be doing. I’ll write more about my processes as I work them out.

I’ll leave this blog post at that, as it’s grown to quite a length. I’m back to work from tomorrow morning, and this afternoon I need to finishing making the spare room into a suitable working environment.

Catch you online soon.

Fundraising testing update

I wrote a post over on fundraising.mozilla.org about our latest round of optimization work for our End of Year Fundraising campaign.

We’ve been sprinting on this during the Mozilla all-hands workweek in Portland, which has been a lot of fun working face-to-face with the awesome team making this happen.

You can follow along with the campaign, and see how were doing at fundraising.mozilla.org

And of course, we’d be over the moon if you wanted to make a donation.

IMG_0373

These amazing people are working hard to build the web the world needs.

Will our latest donation form help us raise more money this year?

DonationDistributionI posted to the fundraising.mozilla.org blog today:

#DALMOOC structure

I hesitantly post this, as I’m spending the evening looking at DALMOOC and hope to take part, but know I’m short on free time right now (what with a new baby and trying to buy a house) and starting the course late.

This is either the first in a series of blog posts about this course, or, we shall never talk about this again.

The course encourages open and distributed publishing of work and assessments, which makes answering this first ‘warm-up’ task feel like more of a commitment to the course than I can really make. But here goes…

Competency 0.1: Describe and navigate the distributed structure of DALMOOC, different ways to engage with peers and various technologies to manage and share personal learning.

DALMOOC offers and encourages learning experiences that span many online products from many providers but which all connect back to a core curriculum hosted on the edX platform. This ranges from learning to use 3rd party tools and software to interacting with peers on commercial social media platforms like Twitter and Facebook. Learners can pick the tools and engagement best suited to them, including an option to follow just the core curriculum within edX if they prefer to do so.

It actually feels a lot like how we work at Mozilla, which is overwhelming and disorientating at first but empowering in the long run.

Writing this publically, however lazily, has forced me to engage with the task much more actively than I might have just sitting back and watching a lecture and answering a quiz.

But I suspect that a fear of the web, and a lack of experience ‘working open’ would make this a terrifying experience for many people. The DALMOOC topic probably pre-selects for people with a higher than average disposition to work this way though, which helps.

Learning about Learning Analytics @ #Mozfest

If I find a moment, I’ll write about many of the fun and inspiring things I saw at Mozfest this weekend, but this post is about a single session I had the pleasure of hosting alongside Andrew, Doug and Simon; Learning Analytics for Good in the Age of Big Data.

We had an hour, no idea if anyone else would be interested, or what angle people would come to the session from. And given that, I think it worked out pretty well.

la_session

We had about 20 participants, and broke into four groups to talk about Learning Analytics from roughly 3 starting points (though all the discussions overlapped):

  1. Practical solutions to measuring learning as it happens online
  2. The ethical complications of tracking (even when you want to optimise for something positive – e.g. Learning)
  3. The research opportunities for publishing and connecting learning data

But, did anyone learn anything in our Learning Analytics session?

Well, I know for sure the answer is yes… as I personally learned things. But did anyone else?

I spoke to people later in the day who told me they learned things. Is that good enough?

As I watched the group during the session I saw conversations that bounced back and forth in a way that rarely happens without people learning something. But how does anyone else who wasn’t there know if our session had an impact?

How much did people learn?

This is essentially the challenge of Learning Analytics. And I did give this some thought before the session…

IMG_0184

As a meta-exercise, everyone who attended the session had a question to answer at the start and end. We also gave them a place to write their email address and to link their ‘learning data’ to them in an identifiable way. It was a little bit silly, but it was something to think about.

This isn’t good science, but it tells a story. And I hope it was a useful cue for the people joining the session.

Response rate:

  • We had about 20 participants
  • 10 returned the survey (i.e. opted in to ‘tracking’), by answering question 1
  • 5 of those answered question 2
  • 5 gave their email address (not exactly the same 5 who answered both questions)

Here is our Learning Analytics data from our session

Screen Shot 2014-10-30 at 13.53.26

Is that demonstrable impact?

Even though this wasn’t a serious exercise. I think we can confidently argue that some people did learn, in much the same way certain newspapers can make a headline out of two data points…

What, and how much they learned, and if it will be useful later in their life is another matter.

Even with the deliberate choice of question which was almost impossible to not show improvement from start to end of the session, one respondent claims to be less sure what the session was about after attending (but let’s not dwell on that!).

Post-it notes and scribbles

If you were at the session, and want to jog your memory about what we talked about. I kind-of documented the various things we captured on paper.

Screen Shot 2014-10-30 at 14.40.57

Click for gallery of bigger images

Into 2015

I’m looking forward to exploring Learning Analytics in the context of Webmaker much more in 2015.

And to think that this was just one hour in a weekend full of the kinds of conversations that repeat in your mind all the way until next Mozfest. It’s exhausting in the best possible way.

Mozilla Contributor Analysis Project (Joint MoCo & MoFo)

I’m  back at the screen after a week of paternity leave, and I’ll be working part-time for next two weeks while we settle in to the new family routine at home.

In the meantime, I wanted to mention a Mozilla contributor analysis project in case people would like to get involved.

We have a wiki page now, which means it’s a real thing. And here are some words my sleep-deprived brain prepared for you earlier today:

The goal and scope of the work:

Explore existing contribution datasets to look for possible insights and metrics that would be useful to monitor on an ongoing basis, before the co-incident workweek in Portland at the beginning of December.

We will:

  • Stress-test our current capacity to use existing contribution data
  • Look for actionable insights to support Mozilla-wide community building efforts
  • Run ad-hoc analysis before building any ‘tools’
  • If useful, prototype tools that can be re-used for ongoing insights into community health
  • Build processes so that contributors can get involved in this metrics work
  • Document gaps in our existing data / knowledge
  • Document ideas for future analysis and exploration

Find out more about the project here.

I’m very excited that three members of the community have already offered to support the project and we’ve barely even started.

In the end, these numbers we’re looking at are about the community, and for the benefit of the community, so the more community involvement there is in this process, the better.

If you’re interested in data analysis, or know someone who is, send them the link.

This project is one of my priorities over the following 4-8 weeks. On that note, this looks quite appealing right now.

So I’m going make more tea and eat more biscuits.

“Conclusions”

Mile long string of baloons (6034077499)

  • Removing the second sentence increases conversion rate (hypothesis = simplicity is good).
  • The button text ‘Go!’ increased the conversion rate.
  • Both variations on the headline increased conversion rate, but ‘Welcome to Webmaker’ performed the best.
  • We should remove the bullet points on this landing page.
  • The log-in option is useful on the page, even for a cold audience who we assume do not have accounts already.
  • Repeating the ask ‘Sign-up for Webmaker’ at the end of the copy, even when it duplicates the heading immediately above, is useful. Even at the expense of making the copy longer.
  • The button text ‘Create an account’ works better than ‘Sign up for Webmaker’ even when the headline and CTA in the copy are ‘Sign up for Webmaker’.
  • These two headlines are equivalent. In the absence of other data we should keep the version which includes the brand name, as it adds one further ‘brand impression’ to the user journey.
  • The existing blue background color is the best variant, given the rest of the page right now.

The Webmaker Testing Hub

If any of those “conclusions” sound interesting to you, you’ll probably want to read more about them on the Webmaker Testing Hub (it’s a fancy name for a list on a wiki).

This is where we’ll try and share the results of any test we run, and document the tests currently running.

And why that image for this blog post?

Because blog posts need and image, and this song came on as I was writing it. And I’m sure it’s a song about statistical significance, or counting, or something…

Something special within ‘Hack the snippet’

Here are a couple of notes about ‘Hack the snippet‘ that I wanted to make sure got documented.

  1. It significantly changed peoples’ predisposition to Webmaker before they arrived on the site
  2. Its ‘post-interaction’ click-through-rate was equivalent to most one-click snippets

Behind these observations, something special was happening in ‘Hack the snippet’. I can’t tell you exactly what it was that had the end-effect, but it’s worth remembering the effect.

1. It ‘warmed people up’ to Webmaker

  • The ‘Hack the snippet’ snippet
    • was shown to the same audience (Firefox users) as eight other snippet variations we ran during the campaign
    • had the same % of users click through to the landing page
    • had the same on-site experience on webmaker.org as all the other snippet variations we tested (the same landing page, sign-up ask etc)
  • But when people who had interacted with ‘Hack the snippet’ landed on the website, they were more than three times as likely to signup for a webmaker account

Same audience, same engagement rate, same ask… but triple the conversion rate (most regular snippet traffic converted ~2%, ‘Hack the snippet’ traffic converted ~7%).

Something within that experience (and likely the overall quality of it) makes the Webmaker proposition more appealing to people who ‘hacked the snippet’. It could be one of many things: the simplicity, the guided learning, the feeling of power from editing the Firefox start page, the particular phrasing of the copy or many of the subtle design decisions. But whatever it was, it worked.

We need to keep looking for ways to recreate this.

Not everything we do going forwards needs to be a ‘Hack the snippet’ snippet (you can see how much time and effort went into that in the bug).

But when we think about these new-user experiences, we have a benchmark to compare things too. We know how much impact these things can have when all the parts align.

2. The ‘post-interaction’ CTR was as good as most one-click snippets

This is a quicker note:

  • Despite the steps involved in completing the ‘Hack the snippet’ on page activity, the same total number of people clicked through when compared to a standard ‘one-click’ snippet.
  • We got the same % of the audience to engage with a learning activity and then click through to the webmaker site, as we usually get just giving them a link directly to Webmaker
    • This defies most “best practice” about minimizing number of clicks

Again, this doesn’t give us an immediate thing we can repeat, but it gives us a benchmark to build on.