Pod1 is a Creative Digital Agency with offices in London and New York. We're the UK's fastest growing Web design & build Agency and we deliver outstanding Online Marketing campaigns and provide excellent Online Strategies.


 
Content
July 12, 2008 by david

A/B Testing - Strawberries or Cherries?

An A/B tested pancake

Here at Pod1 we are very passionate about things and this goes as well for such dry things as A/B testing. Everyone talks about it but what is it and how can you do it?

With A/B testing you take the HiPPO (Highest Paid Person in the Organisation) out of the equation. HiPPOs are usually the ones who say things like: I like this so you have to do it.
A/B testing puts figures behind guesswork and these usually impresses the HiPPO.

A/B testing as well means that you will produce a loosing version and this might lead to lower conversion (if you test conversion). But that is the whole point, as you want to find the winner.

As you may see I can get quite passionate about these things and I tend to take this home with me. This can lead to my Girlfriend being quite upset because after the theatre when she has to listen to me ranting about the Train Ticket machine and its poor usability. I bet Southern Trains has such a high AOV because their interface is so bad.

So in order to make up for that I decided to do, you can guess it, some A/B testing at home. What at home? Yes.

To make up for me always talking about Usability and User experience I decided to make a Pancake for Breakfast for her following the classic test setup.

1. Define a need
Girlfriend hungry.

Good that done we need to define our target audience.

2. Define Target audience:
Just Girlfriend is like “ABC 25-35 with lots of spare cash” (yours in this case), too generic. We need to define the persona: She really likes warm breakfast but also Yoghurt with Fruit, but she doesn’t like muesli very much. Often people only look at the first most obvious definition (ABC 25-35 likes warm breakfast) if they dive into the whole persona development thing, but there is a danger here:
If you serve her a Fry Up (warm breakfast), you probably haven’t understood your target audience very well as the result is likely that she moves back to her Mom.

3. Define success:
The pancake is the base of all and won’t change, it’s like the outer frame of your website unless you test the outer frame of your website. You have to plan what the filling for the pancake will be and make an educated guess about which one will win. This way you define the goals of this test and you can measure the results against it. Careful here, there are many micro actions that look like success (she making noises of appreciation like MMMMM, Yam etc,) but at the end you want the plate to be empty or a 1-2-1 in a very private area of the house. You define the goals so aim high but not too high.

4. Run the test:
Make the pancake and serve it straight from the pan no filling; with this please. Now you need to wait and see how the pancake is perceived. We need to establish a control result to test our variations against. Note down the feedback she gives. If she runs straight to the Jam/Nutella/Sugar Jar, you have done some usability testing at the time. That’s the great thing about testing you do many things at the same time which saves time and money.

After this we set up the test variations. My Girlfriend likes Strawberries and Cherries so I decided to use these. The test took a while to complete but I think I have found the winning combination. But you need to be careful with the initial results.

You will get some feedback very quickly (Yam or Yak) but wait until she is finished. If she doesn’t finish the pancake then even if she said it tastes nice initially, you have done something wrong and you need to test a different combination. Careful with trying to do it with too much fruit (think rich media). Results are initially very encouraging but if you get this wrong the result is a lot of soggy pancake on plate.

As with all good things you need to keep testing (I’m doing that now for about half a year). Sometimes you will get it wrong (Strawberries + Yoghurt and no sugar) and sometimes you will get it right but always keep testing.

Once you are done testing the pancakes you can test the next thing, I’m currently working on Brioche, freshly made.

Pancake with Yoghurt and Cherries

10-15 fresh Cherries
1 tbs granulated sugar
Shot glass of water

75g plain flour
50 ml full fat milk
1g baking powder
1 Egg

3 tbs Greek Yoghurt

Pit the cherries and place in a very small saucepan. Sprinkle with the sugar and put aside for 10 minutes. Place the saucepan on the hob and heat very gently. After 5 minutes add the water and keep on low heat for a little while. The aim is to concentrate the cherry flavour.
While the cherries reduce, mix the flour with the baking powder and add milk and the egg. Use a whisk to make a smooth not too runny pancake dough.

Place the butter in a frying pan and fry the pancake from both sides until golden brown.

Take it out of the pan and place the Yoghurt in a line across the centre, top with the Cherries and flap over in the middle. Serve immediately.

Variations:
Strawberries with a bit of Cognac
Strawberries raw but with Honey on the pancake
Banana with Honey and cinnamon

Multivariate option:
Cherries and strawberries combined
Fry 1 pancake the day before, then cut it in small stripes and fry again with lots of sugar in the pan (Caramelise it basically). Use this as a crunchy topping on the Fruit/Yoghurt variant.

1 Comment »

  1. Yam = Yum, Yak = Yuk

    Right?

    very funny david. so did you end up havign a 1-2-1 in a private place in the house???

    Comment by Fadi — July 14, 2008 @ 10:30 am

RSS feed for comments on this post. TrackBack URI

Leave a comment





Categories:

Search this blog

Archives