When running a startup, never just "see what happens"

If you run an experiment to see what happens, you'll always succeed...in seeing what happens.

But you'll never learn anything!

We haven't defined what success or failure looks like, and thus haven't given any thought to how the outcome of the experiment could change anything.

We have to give ourselves the opportunity to say, "well, that didn't work." If we don't, we can always rationalize any result. We're avoiding all "pivot or persevere" moments and just...walking aimlessly.

That's because experiments aren't valuable unless they're testing an hypothesis.

Think back to that biology or chemistry class: an hypothesis is a single assumption that meets three criteria:

1. It's defined very clearly.

2. It's testing one thing.

3. It's falsifiable.

Any experiment requires us to say in advance what we're trying to DISPROVE — the who, what, how, and when — and the specific way we'll know if we've disproved our hypothesis.

So when we run an experiment, we come up with a measurement that has a clear go/no-go.

Pass or fail.

We define one signal that we can look at that will tell us if we're heading in the right direction.

Examples:

• The click-through rate on our test ad

• How many customers agree to get on a call with us

• The sign up rate from our landing page

• etc.

It's not difficult to pick out WHAT to measure. But how do we know what the measurement should be? How many signups signals success?

We don't always know, and this is where we tend to just "see what happens".

Don't.

Just because we don't know what the result should be ideally (whatever that even means), doesn't mean we can't base it on anything.

Here's how:

Any of these metrics exists as part of a system.

For example, clicking on the ad will eventually lead to landing page conversions, which will eventually lead to freemium-to-premium conversions, which will eventually lead to referrals, etc.

Think of the entire ecosystem in which what we're trying to test sits. Develop a model for all of it.

They're all guesses. You're wrong about all of them.

BUT

By basing it on something — particularly on a whole system — any experiment failure will FORCE us to ask:

• What does it mean to be wrong?

• What are the implications?

• How does this inform our model?

• Does this mean this is less desirable, less viable, or less feasible than we thought?

• What's the right next experiment?

The answers to those questions are far more important than the result of the experiment itself.

TL;DR — never just "see what happens".

1. Form a discrete, specific, falsifiable hypothesis

2. Run an experiment to test it.

3. What does this result mean?