How to measure progress pre-MVP?

When you don't have an MVP with KPIs, what should you look out for?

In this edition

  • How to measure progress for pre-MVP startups

  • A mental model of how your idea develops

  • Concrete signals that help you identify progress

A big welcome to the huge amount of new subscribers! Let's go!

How to measure progress pre-MVP?

What should you measure if you don't have an MVP or business-oriented KPIs? Eric Ries talks 'Build, Measure, Learn', but if you don't have anything to build, how do you measure progress forward?

In this edition, I will give you a mental model that will help you to look for the right signals. Based on my experience with 100+ startups, I often see two distinct phases in an early-stage startup: 'Defining it' and 'refining it'. These two phases tell you what to look out for.

Defining it

You enter the ‘defining’ phase the moment you start. In this phase, you are figuring out what 'it' actually is you do. Here you are hunting for problems, trying to see how you might add value. Most of the time, people overestimate how well they understand their own ideas. Proper alignment with your team is key: talk about your views on your product, your solution, your startup, the problem, your market, since there is a lot to figure out. This is extremely fuzzy and abstract, meaning it can feel like you are swimming around without a sense of direction. Not everyone enjoys being in this phase: people will sometimes decide for the sake of it. Not inherently bad, you’ll soon enough learn if it was right if you listen correctly.

Or you have one of these experiences: all of a sudden, boop, you see 'it'. In design and innovation science it's called a creative leap. In literature on scientific discovery, a scholar named William Whewell in 1840 called these sudden jumps 'happy thoughts' and I find that adorable.

No accident

People often attribute serendipity or randomness to this. Whewell didn't like this:

"It is, [..], only the spark which discharges a gun already loaded and pointed; and there is little propriety in speaking of such an accident as the cause why the bullet hits its mark."

Meaning, these happy thoughts or leaps can only happen if you already have your ammunition lined up and aimed in a certain direction. They are no accidents. For me, ammunition is what you get from customers, reading (anything, I'll show you next week), and being curious about the context you want to enter.

The man responsible for us drinking a lot of milk and not dying of it, Louis Pasteur, wisely said: "Chance favours the prepared mind." If you can't see 'it' yet, you should get more ammunition. Bear in mind: People collect insights at a different rate. Some people can get more from interviews, for instance. That's why the question: "How many interviews are enough" is hard to answer.

To round off this quotation frenzy, I will drop one last of famous Dutch philosopher Johan Cruijf, who happened to play football. "Je gaat het pas zien als je het door hebt", loosely translated as "You will only start to see it once you get it". To make things unfair, some people are better at seeing 'it', meaning they require less input. For example, Person A's happy thought might occur after 20 interviews while for the other person it takes just 5. It's connecting dots, finding patterns, as described in the swiss-army-knife post.

What you should 'measure'?

Startups don't always have the luxury of extensive measurements. When I worked at an online marketplace (Werkspot within the HomeAdvisor family), I was so pleased that there was so much data and NPS scores available to analyse and make inferences about. And sure, when your startup has an MVP that performs at a certain metric, it is easy to have an idea of how well it's going.

However, what if you are not there yet? What if you are still defining 'it'. What is it that you should measure? I think measuring is a misleading term. It's more a question of: what should you look out for when building something people want? I aggregated signals, good and bad, for both the defining and refining phase, based on the 100+ startups I coached.

What to look out for while defining it?

Concept Zero: when you understand 'it'

Concept Zero is a milestone where you can define 'it' properly. This is not the same as product-market fit. Product-market fit is often defined as having the right product so that you can deploy growth mechanics. Concept Zero is the starting point of iterating towards product-market fit.

When do you have concept zero? When you can articulate it to someone else. When you can explain what 'it' is, and what 'it' does and how 'it' works and the customer gets it. Language is a beautiful design tool in that sense. Obviously, you can use other ways, such as mockups to see if the customer gets it. If you can't explain 'it', you might be on to something but you are not there yet.

It is a term I borrowed from pandemics: patient zero. Patient zero is the person we can trace back all infections to. Similarly, concept zero is the concept we can trace all versions back to. Anything before this doesn't bear enough resemblance to this concept. It's a soup, a mess of data, insights, and semi-developed ideas. It's supposed to be like that. However, after a while, you can distill what concept zero is, also by the exclusion of all the options that it is not. To define is to limit (Oscar Wilde).

Everything after Concept Zero are incremental adjustments that build in the same basic premises. You can have many concept zeros in your startup. But most probably only one concept zero's successor is still alive.

Refining it

This phase is distinctly different from the 'defining it'. I see this still as before product-market fit. It is when your 'it' is explainable to your customers. You are now on a way to discovering your beachhead market, from where you can start iterating towards product-market fit.

Showing it

You can now iterate, and the big difference is: You can show 'it' to customers. You can gather feedback on your idea and improve onward. You get much better-directed feedback than during the defining phase, where you were fishing with quite a broad net to understand the customer in general.

It is here that a Build→Measure→Learn (BML) loop might come in handy, it resembles much of Ries' innovation accounting. However, one study showed that only 36% of digital startups report using the BML-loop. It might be harder to implement than on paper as one interviewee reports: "these apparently pragmatic tools sometimes work great in theory but not so well in practice, and did they make us sweat!"

Detailed experiments

Furthermore, once you know 'it', you can run much more detailed experiments. Besides showing it, such as landing pages and brochures to get qualitative feedback, you can build an MVP of 'it' and see how it performs. How much value it generates. You can probably measure quite concretely how well 'it' works. These generally are things you cannot measure while you are defining 'it'.

The picture above is very extreme, transitions are not that clear cut, in reality, you could experience a more smooth transition. However, you probably can point to moments in time where you clearly didn't have a clue, where you clearly were still defining it. And probably you will have a moment in time where you knew quite clearly what 'it' was. Ask yourself: are we in a defining or refining phase for a certain problem or solution? Because that tells us what you should be looking out for, which I will cover in the next section.

What to look for while refining it

Measures are not the holy grail

Neither of these signals is conclusive. As you can see, most of the measures above are customer-centric. I put my focus there since that is what most often lacks, especially in tech/engineering-heavy teams. These measures should aid you to get your bearings. Don't make them the end goal. Tiffani Bova (Author of 'Growth IQ') shared two insightful rules of thumb with a on-point meme:

Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.”

Campbell’s Law: "The more a metric is used, the more likely it is to corrupt the process it is intended to monitor.”

Jeroen's take

Product market-fit is not a goal. Saying you want to reach product-market fit is like saying you want to reach the 21km mark in a marathon. Having more concrete goals will help you to make decisions. The above models should give some insight into what type of goals and measures you could deploy. Only deploy them when your entire team commits to it and also accepts negative feedback. In my failed startup, I did various types of testing but was not actually interested in the answers. I was rushing towards launching, only to fail a couple of months later.

Bear in mind that everything here is just one of many explanations. It's a peek into my personal mental models. These mental models might have quite some experiences behind them, yet there is also scientific proof that I'm oversimplifying things. I'll touch upon that in a later edition. I took some inspiration in visualizing the concept development from this paper by Thomas Ding (2019).

Do you have other signals that you look out for? Let me know in the comments!

Leave a comment

Three content tips

  1. Learn how Twilio founder listened to customers in this podcast (20 minute VC)

  2. Articulate your story better. Write short and to the point. This short article will show you how.

  3. Scaling from 10 to 10.000 employees? A summary of the "High Growth Handbook" Read here

How would you rate this week's newsletter?

Great - Good - Meh

This helps me to get make content for you!

Who should know about this too?

Share Abstract