Tag: actionable metrics

Being Awesome, Going Forth, Innovation, Innovation Mindsets, Lean, Lenses

Charging Up the Wrong Hill

Posted on by GFandBA

I was enjoying a morning run while listening to an excellent game design podcast, Ludology. In episode 113, Geoff Engelstein discussed the Sunk Cost Fallacy. It works like this:

You’ve played a boardgame for a little while with your friends when you realize no one is having any fun. You turn to your friends and say “Well, we’ve gone this far lads. Might as well see it to the end.” Then you proceed to spend an evening trudging through a less-than-enjoyable experience, just because you did not want to waste the time you had already sunk into it. Instead of stopping after wasting one hour on something, you decide to spend another hour on it just to finish it, essentially spending twice as much for no reward.

It seems silly here, but it happens often in innovation.

Ed Catmull uses a model of two hills in his book, Creativity, Inc.

“People need to be wrong as fast as they can. In a battle, if you’re faced with two hills and you’re unsure which one to attack, the right course of action is to hurry up and choose. If you find out it’s the wrong hill, turn around and attack the other one.” – Ed Catmull

He goes on to say that the only wrong attack, is to go between the hills. The Sunk Cost Fallacy would have your squad start attacking one hill, realize the enemy is on the other hill, but continue to charge up this hill because they already made it part of the way up. “We don’t want to waste that initial charge, sarge.”

The Sunk Cost Fallacy is the enemy to innovation. To understand how to defeat it, well use the Lean Startup principle, the Build-Measure-Learn feedback loop, as a lens.

Build

This is the phase that lets you minimize your sunk cost. Keeping costs low helps you mitigate the opportunities for you to say “We’ve spent so much already on this… we should push forward.” A great way to minimize sunk cost is to prototype as minimally as possible. What are you testing? What’s the cheapest way to validate it? Maybe there is a way to build a Paper Prototype or to smokescreen your prototype? Do not splurge on additional features, functionality, or looks. You just need what is minimally viable.

Measure

Numbers never lie, but we tend to bend their truths from time to time. Set the success metrics for your prototype as early as you can. Do not budge on these numbers and be honest with yourself. If you set a goal of a 5% conversion rate, your test may be successful, but are you really reaching for where you need to go? Be wary of vanity metrics. Focus on the stuff that is going to matter.

For a great model of this, look to baseball and the rise of Sabermetrics. Some statistics are easy to calculate in baseball; like batting average or earned runs average. However, some people started to look at new statistics, ones that really highlighted the value of the player towards creating a winning team. Sabermetrics includes stats like runs created or wins above replacement. You need to find the mechanic or action that your prototype needs to accomplish now, and build a statistic tied directly to that, in its most simplistic form.

Learn

Charging up the wrong hill is ok, as long as when you realize it is the wrong hill, you don't keep charging up because that's what you've always done. Time to find the next hill.
Charging up the wrong hill is ok, as long as when you realize it is the wrong hill, you don’t keep charging up because that’s what you’ve always done. Time to find the next hill.

At this point you’ve tested your prototype, collected your success metric data, and your hypothesis has either succeeded (in which case, ignore me) or it has failed (read on, dear reader!). This is the exciting part. You’ve proven, with numbers, that you’ve charged up the wrong hill and it is time to face facts. Now you get to pivot! This might mean you need to spend more time understanding and building empathy for your customer. This might mean you need to evaluate the constraints of your test. Perhaps your solution has driven too far from the problem. There is a world of opportunity and learning at a pivot point… AS LONG AS YOU DONT KEEP CHARGING UP THE HILL!

But I mean, why would you? You kept your costs low by building a minimally functioning prototype, you set your success metrics early, and you failed/learned. This is a great moment! Celebrate as you get out your tactical map, cross the hill off with a red marker, and proclaim “Time to charge up that other hill!”

Challenge

Has there ever been a time when you charged up a hill even after you new it was the wrong hill?

How did the second half of that charge feel?

What are ways to minimize your sunk cost?

0
Failure, Going Forth, Innovation

Vanity Metrics & Bathrooms: It’s not the size of your mirror.

Posted on by GFandBA

Numbers never lie. However numbers can be misleading, even when they don’t mean to be.

One thing that my good friend Joe Greaser has often said “If someone is bragging about their data, it is probably a bunch of vanity metrics.”

Eric Reis (Lean Startup) brought the term “vanity metric” to light. They are the metrics that sound really good, but in the end don’t amount to much. In contrast to vanity metrics are the actionable metrics. These metrics are tied to specific actions or tests and inform you on what to do with them. “My app has had 50,000 downloads.” Vanity metric. “I tested two versions of my app and customers purchased 40% more through the B version.” Actionable metric.

Another one of my good friends, Mike Jarrell, was going to a sporting event where they were promoting stadium upgrades. “87% more female restroom facilities and 20% more male facilities!” At first glance you say “Wow! They really understand that lines for the women’s restroom at sporting events can be painfully long.

But are we falling into the trap of vanity metrics?

Let’s take this to absurdity to prove a point. Assume that Innovation Stadium has 5 female restrooms and 20 male restrooms. Pretty unlikely and unreasonable, I know… but we’re imagining the absurd here. Stick with me. Now let’s agree that Innovation Stadium added 5 more female restrooms and 5 male restrooms. The grand totals are still unbalanced at 10 female restrooms and 25 male restrooms. And yet Innovation Stadium can claim “We’ve added 100% more female restroom facilities and 25% more male facilities.” Sound familiar? Not saying that’s what happened, but sometimes you pick the numbers you want to market.

What sounds better? 5 more female restrooms or 100% more?

Recently, TechCrunch posted about the math behind startup valuations. This is an amazing article, especially for a mathy like me. And I have no doubt that these calculations are reliable within certain parameters. I want to point out my one concern. The article says that without enough customers, a startup has to use estimations. Believe me, I get it. You have to estimate sometimes, especially in a new business. However, if any startup bragged about data calculated from estimates, then they too have been wooed by vanity metrics.

It reminds me about how quark-sized some baseball statistics have become. Almost to the point where a batter can walk to the plate while the announcer tells you his batting average in the month of August, with runners in scoring position, against pitchers with a weekend birthday this year, when they’ve gotten a call from their mom before the game.

Vanity metrics may look good, but they lure you into dangerous assumptions of success.
Vanity metrics may look good, but they lure you into dangerous assumptions of success.

In innovation, we don’t have the kind of time baseball has. They have a 162-game season to see their statistics play out. In innovation, you’re lucky if you have a couple weeks. Build, test, measure, learn… sprint, sprint, sprint. So how do we avoid vanity metrics?

  1. Establish what hypothesis you are testing, before you test.
  2. Identify what metric, or metrics, would absolutely validate your hypothesis.
  3. Now you can test your prototype.
  4. Collect, reflect, observe, and analyze.
  5. Be ok with failing forward.

By setting the hypothesis and success metrics before you test will prevent you from latching on to bright spots. Also, by being alright with failing (as long as you are failing forward) then you feel less pressure for each test to be successful or be validated. It is a tough practice, because you’d like your idea to be a winner, but this is all part of the process in finding the right solution. Remember, you’re refining an idea that will work, not just pushing your favorite to the finish line. So be modest, avoid the vanity metrics, and keep it all actionable.

UPDATE!!!

So here’s something pretty applicable. Early today, another good friend of mine, Adrienne Campbell, and I had a great conversation about this post. I am lucky to have many good friends who:

  1. are deep thinkers
  2. challenge me and make me think deeper

We were talking about vanity metrics and if there really was a good use for them. Maybe they don’t need to be avoided at all costs. Perhaps they could provide some value.

Take a look at this example.

A vanity metric for blogs would be the number of views. This is a great statistic if you are looking for overall exposure and reach. “Should we promote on Blog X? How many views do they get?” A great actionable metric would be “How many excellent conversations came out of a post?” (that’s one from today!) or “When I write about Subject A, how many views do I get compared to when I write about Subject B?” I get more views in WordPress when I write about writing. I get more retweets when I write about innovation processes. This is data I can act on.

Something Adrienne brought up was (and I’m paraphrasing) “Different metrics are appropriate depending on the purpose.”

  • Want to know if your prototype is working? You need to locate actionable metrics to test.
  • Want to promote your solution to an outsider? There may be some vanity metrics that get the conversation going.

It brings it all back to the stadium restroom example. They probably justified the construction based on actionable metrics such as length of wait and restrooms per person, but they promoted vanity metrics by promoting the percentage increase.

That means both can be a welcome tool in your innovative tool box. You just have to know when to use which one. ~GFandBA

2