Tag: testing

User Testing vs. A/B Testing

So you have a page, or a system, or a form or an app or anything, and you know you want to make it ‘better’.

And the question is…

Should we use User Testing or A/B Testing?

TL;DR: Both can be valuable, and you can do both. So long as you have an underlying indicator of the ‘better’ that you can track over the long-term.

A/B Testing

A/B testing is good because:

  • It shows you ‘for real’, at scale, what actually prompts your users to do the things you most want them to do
  • It can further polish processes that have been through thorough user testing; to a degree that’s just not possible with a small group of people
  • It can self-optimize content in real-time to maximize the impact of short spikes of campaign activity

A/B testing is bad because:

  • You need a lot of traffic to get significant results which can prevent testing of new or niche products
  • You can get distracted by polishing the details, and overlook major UX changes that would have significant impact
  • It can slow down the evolution of your project if you try to test *everything* before you ship

User Testing

User testing is good because:

  • You can quickly get feedback on new products even before they are live to the public; you don’t need any real traffic
  • It flags up big issues that the original designers might miss because of over familiarity with their offering (the ‘obvious with hindsight’ things)
  • Watching someone struggle to use the thing you’ve built inspires action (it does for me at least!)

User testing is bad because:

  • Without real traffic, you’re not measuring how this works at scale
  • What people say they will do is not the same as what they do (this can be mitigated, but it’s a risk)
  • If you don’t connect this to some metrics somewhere, you don’t know if you’re actually making things better or you’re stuck in an infinite loop of iterative design

An ideal testing process combines both

  1. For the thing you want to improve, say a webpage, decide on your measure of success, e.g. number of sign-ups
  2. Ensure this measure of success is tracked, reported on and analyzed over time and understand if there are underlying seasonal trends
  3. Start with User Testing to look for immediate problems/opportunities
  4. If you have enough traffic, A/B Test continually to polish your content and design features
  5. Keep checking the impact of your changes (A/B and User Testing) against your underlying measure of success
  6. Repeat User Testing
  7. A/B Testing can include ideas that come from User Testing sessions
  8. Expect to see diminishing returns on each round of testing and optimization
  9. Regularly ask yourself, are the potential gains of another round of optimization worth the effort? If they’re not, move onto something else you can optimize.

That’s simplified in parts, but is it useful for thinking about when to run user testing vs. A/B testing?

Weeknotes 5 – Webmaker Workweek

View from Mozilla Space Toronto
View from Mozilla Toronto

As I’m halfway into the following week I’m writing these notes quickly rather than losing them completely. I apologize in advance 🙂

Week 5 was spent in Toronto with the Webmaker team and it will be hard for a quick write-up to do this week justice. I got to hack and hang-out with about half of the total Mozilla Foundation staff, which is hugely valuable four weeks into a job where you mostly work remotely. IRC handles turned into real people, and the people turned out to be very special. So first, thanks to this amazing team for welcoming me so kindly. I think we crammed a year’s worth of social activity into a week’s worth of evenings and across the whole week, I almost got a whole night’s worth of sleep.

I signed more that one waiver in the name of fun this week
I signed more than one waiver in the name of fun this week

There’s a test that goes something along the lines of “people you wouldn’t mind getting stuck at an airport with”, and everyone I met last week would pass that test. Genuinely.

I thought this week might have been a lot of talking and leaving with too many ideas to implement, but from the start it was structured to create measurable output.

Sunday in the office, the bugzilla tickets were transformed into a physical scrum board:

Webmaker Scrum Board

And during the week, these discrete tasks moved from To Make > Making > Made

In the metrics track, I was lucky to work closely with Scott Downe who taught me a tonne of useful things, and we shipped some stuff too. Including a brand new process to make continual testing and optimisation of the Webmaker tools practical.

You can see the new testing process here, and our tests and the results will be open for you to follow along as we learn more about our tools and how people use them.

Onwards…

I like this Canadian alternative to the UK's "One Way"
I like this Canadian alternative to the UK’s “One Way”

And I would be negligent not to include the gif of the week:

Something I wrote for Engaging Networks

A few weeks ago I received a marketing email from the Engaging Networks team quoting some stats about the possible improvements to website conversion rates that can be achieved with A/B testing.

I was caught off guard (but pretty chuffed) when I realised I was being quoted my own case study from a presentation I had given a couple of years earlier.

I sent a quick reply to the email and was delighted to find it was sent from a real address with a real person at the other end reading the replies (@Rachel_shares).

This turned into a nice discussion about conversion rate testing, and somehow I agreed to write a guest blog post. Which, with some helpful editing from Rachel has now been posted on the Engaging Networks blog.

I thought I should share the link with all two of my readers on this blog, just in case you’re not also reading the EN blog 🙂

The real story behind WWF’s fundraising split test success:
http://www.engagingnetworks.net/uk/blog/wwf-split-testing