Numbers never lie. However numbers can be misleading, even when they don’t mean to be.

One thing that my good friend Joe Greaser has often said “If someone is bragging about their data, it is probably a bunch of vanity metrics.”

Eric Reis (Lean Startup) brought the term “vanity metric” to light. They are the metrics that sound really good, but in the end don’t amount to much. In contrast to vanity metrics are the actionable metrics. These metrics are tied to specific actions or tests and inform you on what to do with them. “My app has had 50,000 downloads.” Vanity metric. “I tested two versions of my app and customers purchased 40% more through the B version.” Actionable metric.

Another one of my good friends, Mike Jarrell, was going to a sporting event where they were promoting stadium upgrades. “87% more female restroom facilities and 20% more male facilities!” At first glance you say “Wow! They really understand that lines for the women’s restroom at sporting events can be painfully long.

But are we falling into the trap of vanity metrics?

Let’s take this to absurdity to prove a point. Assume that Innovation Stadium has 5 female restrooms and 20 male restrooms. Pretty unlikely and unreasonable, I know… but we’re imagining the absurd here. Stick with me. Now let’s agree that Innovation Stadium added 5 more female restrooms and 5 male restrooms. The grand totals are still unbalanced at 10 female restrooms and 25 male restrooms. And yet Innovation Stadium can claim “We’ve added 100% more female restroom facilities and 25% more male facilities.” Sound familiar? Not saying that’s what happened, but sometimes you pick the numbers you want to market.

What sounds better? 5 more female restrooms or 100% more?

Recently, TechCrunch posted about the math behind startup valuations. This is an amazing article, especially for a mathy like me. And I have no doubt that these calculations are reliable within certain parameters. I want to point out my one concern. The article says that without enough customers, a startup has to use estimations. Believe me, I get it. You have to estimate sometimes, especially in a new business. However, if any startup bragged about data calculated from estimates, then they too have been wooed by vanity metrics.

It reminds me about how quark-sized some baseball statistics have become. Almost to the point where a batter can walk to the plate while the announcer tells you his batting average in the month of August, with runners in scoring position, against pitchers with a weekend birthday this year, when they’ve gotten a call from their mom before the game.

In innovation, we don’t have the kind of time baseball has. They have a 162-game season to see their statistics play out. In innovation, you’re lucky if you have a couple weeks. Build, test, measure, learn… sprint, sprint, sprint. So how do we avoid vanity metrics?

1. Establish what hypothesis you are testing, before you test.
2. Identify what metric, or metrics, would absolutely validate your hypothesis.
3. Now you can test your prototype.
4. Collect, reflect, observe, and analyze.
5. Be ok with failing forward.

By setting the hypothesis and success metrics before you test will prevent you from latching on to bright spots. Also, by being alright with failing (as long as you are failing forward) then you feel less pressure for each test to be successful or be validated. It is a tough practice, because you’d like your idea to be a winner, but this is all part of the process in finding the right solution. Remember, you’re refining an idea that will work, not just pushing your favorite to the finish line. So be modest, avoid the vanity metrics, and keep it all actionable.

UPDATE!!!

So here’s something pretty applicable. Early today, another good friend of mine, Adrienne Campbell, and I had a great conversation about this post. I am lucky to have many good friends who:

1. are deep thinkers
2. challenge me and make me think deeper

We were talking about vanity metrics and if there really was a good use for them. Maybe they don’t need to be avoided at all costs. Perhaps they could provide some value.

Take a look at this example.

A vanity metric for blogs would be the number of views. This is a great statistic if you are looking for overall exposure and reach. “Should we promote on Blog X? How many views do they get?” A great actionable metric would be “How many excellent conversations came out of a post?” (that’s one from today!) or “When I write about Subject A, how many views do I get compared to when I write about Subject B?” I get more views in WordPress when I write about writing. I get more retweets when I write about innovation processes. This is data I can act on.

Something Adrienne brought up was (and I’m paraphrasing) “Different metrics are appropriate depending on the purpose.”

• Want to know if your prototype is working? You need to locate actionable metrics to test.
• Want to promote your solution to an outsider? There may be some vanity metrics that get the conversation going.

It brings it all back to the stadium restroom example. They probably justified the construction based on actionable metrics such as length of wait and restrooms per person, but they promoted vanity metrics by promoting the percentage increase.

That means both can be a welcome tool in your innovative tool box. You just have to know when to use which one. ~GFandBA