Let's face it, when growing demand for your business, you're faced with a bunch of uncertainties.
Can I hire the talent needed to succeed? Will I be able to generate enough demand for my product to be profitable? Are the marketing channels I'm dependent upon for growth going to be too competitive? Will I have to learn TikTok; I'm not funny enough :/
Listen to the podcast episode here:
iTunes
Stitcher
SpotifyHowever, other things have a little more certainty around them. Most successful businesses master a small number of marketing channels that drive a disproportionate amount of their growth. Only a small number of the experiments you run will result in something meaningful for your company. And the channels that drive growth today will eventually plateau in the future.
A marketer focused on growing demand for their business is much like a portfolio manager. That portfolio is comprised of assets in the form of marketing channels:
- Those assets need to grow over time.
- The portfolio needs rebalancing over time by discovering new assets to drive future growth.
- Experiments help the marketer discover those assets.
Unfortunately, unlike a portfolio manager, if the marketer f*cks things up, the Fed won't swoop in and artificially boost their numbers, but, a story for a different day :)
Let's talk about why it's essential to understand the risk vs. rewards of experiments as a form of research, the right way to manage a portfolio of experiments, and how to use experiments to diversify your growth channels.
The Risk vs. Return of Experiments
Experiments are an excellent way for marketers and growth professionals to uncover hidden opportunities that will have a considerable impact on their business.
Experiments are just a form of research. Sure they sound way cooler than other types of research - I would prefer to tell people I'm running experiments all day vs. running surveys all day. But, they serve the same purpose. They get you the information and data required to make a decision.
Unlike other types of research, experiments can pose some inherent risk to the business.
What do we mean by risk? Urgh, I didn't know I was risking anything by running experiments!
Well, in this case, the risk involved is composed of three things:
- Time invested (how long it took to gather the research)
- Resources invested (how many resources it took to gather the research)
- Cost invested (how much money you had to spend to gather the research)
All of that adds up to opportunity cost, what else could you have applied that time, resources and cost too.
The opportunity cost for most forms of research is low.
However, with experiments, the range can be anywhere between low and high, depending on how complex they are to run.
Most experiments aim to prove or disprove a belief you have about your users; it's not to predict an outcome for a metric.
For example, one of the more common test marketers run is changing both copy and colors on their website to improve their conversion rate. Their hypothesis would be, if I change the color of the button to green, my conversion rate will increase by 10%. However, if that happens, and you make all your buttons green. You still haven't captured the reason your audience prefers green to other colors. Maybe in certain countries, the color green has different meanings.
Instead, develop the experiment around what you believe to be true of your users, e.g., in country X, the color green is synonymous with a positive decision, so by turning more of our buttons green, more people will click on them.
The amount of risk you're willing to take on with an experiment should be in proportion to the potential of return of the experiment being successful.
Aggressive vs passive experimentation based on lifecycle stage
Early testing = Aggressive | Late testing = Passive
When you’re early, you should be running aggressive tests. Don’t just change the button color, change the whole damn landing page. The upside is more upside, the downside is it’s hard to determine what exactly caused the change.
As you begin getting more quick upside and “wins” you’ll then need to target unique variables in the equation. The upside is confident in the “why did this work?” question but the downside is you’re likely to get smaller step increases.
When you’re early in the testing cycle, run aggressive tests changing many variables. When you’re late in the testing cycle, run more targeted tests looking at only one variable.
Managing a Portfolio of Experiments
The below chart is the perfect method of managing risk. The more risk I take on, the more significant the opportunity for returns, much like a portfolio manager would do when making investments.
Like any good portfolio, most of my experiments are in the low-risk bucket, sure the returns may be lower, but they'll add up over time. I have a couple of experiments at the upper end, where I'm willing to invest more time, resources, and cost because the potential returns are high.
The problems arise when you're chart looks like a version of this:
I'm running too many experiments where the risk to returns doesn't match. Why does that happen? Well, because marketers love to run experiments, and often run experiments for the sake of running experiments, even when the potential rewards are low.
I know, shocking, and I'm sure this isn't you, it's those 'other' marketers.
The other thing that happens is we're running a few experiments, but the majority of them require a lot of time, resources, and cost to get live. Like the portfolio manager who fomo'd into Bitcoin at $14k, waiting for it's the inevitable march to $100k, only to find it drop to $4k, we've taken on a lot of risks, that could be detrimental to our success.
Remember, a small number of experiments will result in something meaningful for your company, so you need to run a lot of experiments to find the few that will make that impact. Running a small amount, with a lot of those being in the high-risk category, isn't a great strategy.
Diversification with Experiments
Ok, I get how I should think about my experiment portfolio, but, why should I care, I'm growing like a rocket, I don't need diversification.
The sad truth about growth is that *most* of the channels helping to grow your company today will start to plateau in the future.
A fun exercise is to have your team do is a saturation exercise, what is the total addressable market for the marketing channels we're growing from, and how much room do we have left for growth? It's not going to be a hundred percent perfect, but it does get you to think about where future growth will come from.
For example, how much available search traffic is there for the keywords you're targeting? How much can you spend on display advertising before you reach diminishing returns? How many affiliates can resell your product and how many sales per affiliate do you expect, and so on.
One way to think about your experiment portfolio is how many of them are trying to improve returns from your core marketing channels, and how many are trying to discover new marketing channels for future growth. As you become more saturated, you're going to invest in more experiments with higher risk as you look to discover new growth opportunities.
Another framework you can base your portfolio on is the horizon framework.
Horizon One -> The marketing channels driving most of our growth. Experiments in here are low risk as we know these channels work.
Horizon Two -> Marketing channels that have demonstrated some kind of growth potential. Experiments are a little riskier as channels are less proven.
Horizon Three -> Speculative bets. The rewards could be great, but a high likelihood of failure.
The vital part is to have some approach to how you continue to grow in existing channels, but leverage experiments to discover new opportunities for future growth.
>> Remember to Listen to the Full Podcast Episode <<
