web design – Adam Lofting

“Conclusions”

Removing the second sentence increases conversion rate (hypothesis = simplicity is good).
The button text ‘Go!’ increased the conversion rate.
Both variations on the headline increased conversion rate, but ‘Welcome to Webmaker’ performed the best.
We should remove the bullet points on this landing page.
The log-in option is useful on the page, even for a cold audience who we assume do not have accounts already.
Repeating the ask ‘Sign-up for Webmaker’ at the end of the copy, even when it duplicates the heading immediately above, is useful. Even at the expense of making the copy longer.
The button text ‘Create an account’ works better than ‘Sign up for Webmaker’ even when the headline and CTA in the copy are ‘Sign up for Webmaker’.
These two headlines are equivalent. In the absence of other data we should keep the version which includes the brand name, as it adds one further ‘brand impression’ to the user journey.
The existing blue background color is the best variant, given the rest of the page right now.

The Webmaker Testing Hub

If any of those “conclusions” sound interesting to you, you’ll probably want to read more about them on the Webmaker Testing Hub (it’s a fancy name for a list on a wiki).

This is where we’ll try and share the results of any test we run, and document the tests currently running.

—

And why that image for this blog post?

Because blog posts need and image, and this song came on as I was writing it. And I’m sure it’s a song about statistical significance, or counting, or something…

One month of Webmaker Growth Hacking

This post is an attempt to capture some of the things we’ve learned from a few busy and exciting weeks working on the Webmaker new user funnel.

I will forget some things, there will be other stories to tell, and this will be biased towards my recurring message of “yay metrics”.

How did this happen?

Screen Shot 2014-09-01 at 14.25.29

As Dave pointed out in a recent email to Webmaker Dev list, “That’s a comma, not a decimal.”

What happened to increase new user sign-ups by 1,024% compared the previous month?

Is there one weird trick to…?

No.

Sorry, I know you’d like an easy answer…

This growth is the result of a month of focused work and many many incremental improvements to the first-run experience for visitors arriving on webmaker.org from the promotion we’ve seen on the Firefox snippet. I’ll try to recount some of it here.

While the answer here isn’t easy, the good news is it’s repeatable.

Props

While I get the fun job of talking about data and optimization (at least it’s fun when it’s good news), the work behind these numbers was a cross-team effort.

Aki, Andrea, Hannah and I formed the working group. Brett and Geoffrey oversaw the group, sanity checked our decisions and enabled us to act quickly. And others got roped in along the way.

I think this model worked really well.

Where are these new Webmaker users coming from?

We can attribute ~60k of those new users directly to:

Traffic coming from the snippet
Who converted into users via our new Webmaker Landing pages

Data-driven iterations

I’ve tried to go back over our meeting notes for the month and capture the variations on the funnel as we’ve iterated through them. This was tricky as things changed so fast.

This image below gives you an idea, but also hides many more detailed experiments within each of these pages.

With 8 snippets tested so far, 5 funnel variations and at least 5 content variables within each funnel we’ve iterated through over 200 variations of this new user flow in a month.

We’ve been able to do this and get results quickly because of the volume of traffic coming from the snippet, which is fantastic. And in some cases this volume of traffic meant we were learning new things quicker than we were able to ship our next iteration.

What’s the impact?

If we’d run with our first snippet design, and our first call to action we would have had about 1,000 new webmaker users from the snippet, instead of 60,000 (the remainder are from other channels and activities). Total new user accounts is up by ~1,000% but new users from the snippet specifically increased by around 6 times that.

One not-very-weird trick to growth hacking:

I said there wasn’t one weird trick, but I think the success of this work boils down to one piece of advice:

Prioritize time and permission for testing, with a clear shared objective, and get just enough people together who can make the work happen.

It’s not weird, and it sounds obvious, but it’s a story that gets overlooked often because it doesn’t have the simple causation based hooked we humans look for in our answers.

It’s much more appealing when someone tells you something like “Orange buttons increase conversion rate”. We love the stories of simple tweaks that have remarkable impact, but really it’s always about process.

More Growth hacking tips:

Learn to kill your darlings, and stay happy while doing it
- We worked overtime to ship things that got replaced within a week
- It can be hard to see that happen to your work when you’re invested in the product
  - My personal approach is to invest my emotion in the impact of the thing being made rather than the thing itself
  - But I had to lose a lot of A/B tests to realize that
Your current page is your control
- Test ideas you think will beat it
- If you beat it, that new page is your new control
- Rinse and repeat
- Optimize with small changes (content polishing)
- Challenge with big changes (disruptive ideas)
Focus on areas with the most scope for impact
- Use data to choose where to use data to make choices
- Don’t stretch yourself too thin

What happens next?

We have some further snippet coverage for the next couple of weeks, but not at the same level we’ve had recently, so we’ll see this growth rate drop off
We can start testing the funnel we’ve built for other sources of traffic to see how it performs
We have infrastructure for spinning up and testing landing pages for many future asks
This work is never done, but with any optimization you see declining returns on investment
- We need to keep reassessing the most effective place to spend our time
- We have a solid account sign-up flow now, but there’s a whole user journey to think about after that
- We need to gather up and share the results of the tests we ran within this process

Testing doesn’t have to be scary, but sometimes you want it to be.

User Testing vs. A/B Testing

So you have a page, or a system, or a form or an app or anything, and you know you want to make it ‘better’.

And the question is…

Should we use User Testing or A/B Testing?

TL;DR: Both can be valuable, and you can do both. So long as you have an underlying indicator of the ‘better’ that you can track over the long-term.

A/B Testing

A/B testing is good because:

It shows you ‘for real’, at scale, what actually prompts your users to do the things you most want them to do
It can further polish processes that have been through thorough user testing; to a degree that’s just not possible with a small group of people
It can self-optimize content in real-time to maximize the impact of short spikes of campaign activity

A/B testing is bad because:

You need a lot of traffic to get significant results which can prevent testing of new or niche products
You can get distracted by polishing the details, and overlook major UX changes that would have significant impact
It can slow down the evolution of your project if you try to test *everything* before you ship

User Testing

User testing is good because:

You can quickly get feedback on new products even before they are live to the public; you don’t need any real traffic
It flags up big issues that the original designers might miss because of over familiarity with their offering (the ‘obvious with hindsight’ things)
Watching someone struggle to use the thing you’ve built inspires action (it does for me at least!)

User testing is bad because:

Without real traffic, you’re not measuring how this works at scale
What people say they will do is not the same as what they do (this can be mitigated, but it’s a risk)
If you don’t connect this to some metrics somewhere, you don’t know if you’re actually making things better or you’re stuck in an infinite loop of iterative design

An ideal testing process combines both

For the thing you want to improve, say a webpage, decide on your measure of success, e.g. number of sign-ups
Ensure this measure of success is tracked, reported on and analyzed over time and understand if there are underlying seasonal trends
Start with User Testing to look for immediate problems/opportunities
If you have enough traffic, A/B Test continually to polish your content and design features
Keep checking the impact of your changes (A/B and User Testing) against your underlying measure of success
Repeat User Testing
A/B Testing can include ideas that come from User Testing sessions
Expect to see diminishing returns on each round of testing and optimization
Regularly ask yourself, are the potential gains of another round of optimization worth the effort? If they’re not, move onto something else you can optimize.

That’s simplified in parts, but is it useful for thinking about when to run user testing vs. A/B testing?

Specific thoughts on the @Coursera experience

First, I’d like to say a massive thank you. I really value the chance to study this excellent material at zero financial cost, and more importantly I love the opportunity you provide to people all around the world who don’t have the finances or the circumstances to otherwise consider such an education.

I also know what it’s like to maintain and develop a complex online system while supporting active users, so this feedback is by no means an accusation of negligence. You will have thought about much of this already I’m sure, and if it’s already on a project roadmap somewhere then please excuse me.

In short, this is not a letter from a grumpy customer; I just thought it may be useful to hear some specific feedback and ideas that could help with the online experience:

When viewing and submitting assignments

Include some visual indicator as to which ones you’ve already submitted. A tick would be plenty.
Likewise for showing which assignments you have completed your peer-reviews for. If you forget, you have to click into each item to check what you have and haven’t done. Even then it’s confusing to remember.
The general visual hierarchy on this page is confusing. Those blue buttons jump out way more than the text you really want people to read (i.e. the assignment titles)
Indicate the assignments where the existing submission deadlines are closed (I’m only in week 2 so maybe this happens after week 1 evaluation is done, but currently its an effort to digest what my next steps are and how much I have to do before Sunday night)

Class Homepage

Bubbling up some top level stats on assignments due/completed to the homepage would be useful

Syllabus page

Ability to mark-off each item you have watched/completed would be nice. Like the assignments, if you’re doing this in the evenings after working, and you’re already tired, every little helps. I found myself relying on visited link colour, and that’s not a very cross-platform solution 🙂

In summary, the simpler you can explain what’s expected of people (and by when), the more enjoyable the learning experience will become. Let them focus on the learning, rather than the admin (unless of course you’re secretly trying to teach personal admin skills).

That’s it for now, as I have homework to do!

I hope that’s useful in some way, and thanks again.

On my next pet project and @Coursera

My most recent ‘pet project’, Done by When, grew up today.

It’s 3 months to the day since I announced a vague plan to test out an idea that had been floating around my head, and now it’s out of beta, taking payments and I’ve just notice our Mandrill email reputation has crept up to ‘Excellent’. Woohoo.

I’m delighted with where it’s going and all the helpful (positive and negative) feedback I’ve had from the first brave group of testers.

I’ve added some screenshots to my portfolio on Behance, but the interface has progressed even further since then.

Now that Done by When has a “business model” and all that, it will be given a serious amount of time and attention going forwards. But importantly, as it has an active user base I won’t be using it as a playground for new ideas and technology. It will first and foremost serve the needs of the users. Which means it’s no longer a ‘pet project’.

I needed another project/playground so I’ve enrolled (and completed my first week) in Design: Creation of Artifacts in Society with Coursera. I’ve studied design before, so mainly wanted to see what the Coursera experience was like in relation to the Open University courses I took a few years back. I’m more interested in the content of the Game Theory course, but that doesn’t start for a while yet, and all learning is good learning.

So I’ll be writing some posts about the Coursera experience, but more importantly I’ll use this as a framework for my next pet project. There are 7 weeks left to go and I’ve set myself the brief to somehow contribute to dealing with the issue of food waste.

Food is core. If we solve food, we solve most things.

Not that I’ll solve food, but I may contribute something.

I’ll keep you posted.

Done by When (beta) is live

So, I just about scraped in inside my (second) deadline.

Done by When is live. Though very much in beta.

Never go live on a Friday: Fail
Ship early: Succeed
Ship often: To be seen

In building this to-do list app I’ve learnt a few new things:

All of which I can highly recommend.

There’s loads more to do. It’s still un-branded for a start and the responsive stylesheets need tidying up, but the basic service offered if fully functional and I think it brings something new to the to-do list marketplace.

Please let me know what you think.

Announcing ‘Done by When’

I promised to ship a new piece of software today but I haven’t quite made it.

Ironically it’s a tool for managing expectations, and visualizing likely delivery times for a given piece of work. It would have been useful!

I hate making excuses, but it’s been a crazy month with lots of good interruptions (lovely clients with interesting projects) and bad interruptions (family emergencies and so on).

So while it’s not ready for you to use today, I’ll have to settle with announcing the project title today, ‘Done by When‘.

A version of the tool, whether it’s ‘finished’ or not will ship by this time next week.

Thought I didn’t make the deadline this time, it’s been very helpful for focussing the mind.

On appropriate design for an appropriate budget

I’ve just launched a simple website for a friend, and in part it was a pleasure to work on because we weren’t trying to re-invent the wheel. All it needed was clean and clear communication and the functionality for her to maintain the site herself.

Only a few years ago, this would have been a messy and much more expensive process, but with open source software as the foundation (WordPress in this case) a small budget can deliver a decent product if you trust your web developer. This is particularly true of location based businesses; Claire is an Osteopath working in Bristol, so doesn’t need the most amazing osteopathy website in the whole world – just a slightly better website than the other osteopaths working in the same area.

In this scenario, good communication is the avoidance of bad communication. This is a subtle but important distinction when trying to communicate on a limited budget. This website won’t win her business, instead it will reduce the potential loss of business that no website, or a badly designed website would have had. Her business will still come from the quality of treatment and patient care she offers.

Even at this ‘entry’ level of web design and build it’s possible to ship quality code. This is a bespoke, HTML5, CSS3, responsive design, and including all configuration, installation, testing, populating, image sourcing etc still came in at just 30 hours work.

It would have taken longer (and cost more) if for example Claire had wanted to debate 6 different types of headline font; and this is the key to appropriate design. We could easily have spent three times as long iterating design concepts, but this would not change the marketplace in which Claire’s messaging needs to operate.

If you’re working on a limited budget, find a developer/team you can trust and talk to them about what you want to achieve. In their hands, your money can go much further.

Web design in a few years time

Link: Web design in a few years time

The more I learn about CSS3, the less I think that photoshop and the ‘design stage’ of current web design practices will be at all relevant in just a few years time. I think that within a year I’ll have stopped using photoshop on any project I was developing for myself. And for work that other people need to review, new practices will to evolve to cope with that. But it’s enough of a change for me to think that it’s not worth upgrading my own copy of photoshop any more. Design will happen in the browser, and for basic photo adjustments, it’s amazing how much you can achieve with something as simple as picassa.

This change is a challenge to everyone who works in web design and I suspect it will leave a few designers, developers and other webbies in limbo.

P.s. The title (and corresponding link) are not to suggest that CSS3 is not valid right now. Only that working processes will take a little time to catch up with this opportunity.