What we cover in Episode 25

How can you build a world-class product team?

We talk to Claire Vo who is the co-founder of Experiment Engine (sold to Optimizely) and now SVP of Product at Optimizley about this very subject.

We cover:

- Why world-class product teams are entrepreneurial, autonomous and focus on customer value.
- How to build a diverse product team, and why that's a great thing
- Why experimentation should be part of your product teams culture

Happy Growing!

1. World-class product teams are entrepreneurial, autonomous and focus on customer value.

The PM's at Optimizely are responsible for the health of the business, not just the health of the products. It means they need to think like both a PM and GM.

To do this, you need to have an entrepreneurial mindset, thrive on risk-taking and ownership, plus be greedy for continual success.

[perfectpullquote align="full" bordertop="" cite="" color="#16989D"]"I think there's a certain characteristic or trait of people who really thrive on ownership and thrive on risk-taking and are maybe a little bit greedy, that brings to an organization that level urgency and ownership that a CEO or founder would bring to their company. It's something I screen for when hiring PMs" - Claire Vo[/perfectpullquote]

Although each product team has a lot of autonomy, they all share the same objectives and metrics through the OKR methodology. They have three key objectives:

- Increase the number of experiments Optimizely customers are running
- Increase customer maturity by helping customers to get better at experimentation
- Build enterprise-grade software that meets the needs of the largest brands in the world.

The objectives are tied back to metrics shared by the entire product team, for example - the sum of ARR (annual recurring revenue) of accounts that are running four or more experiments a month, and how much ARR of their accounts are running server-side experiments.

Wait, what're server-side experiments?

Server-side experiments are where Claire sees a lot of experimentation moving. Instead of focusing on testing different elements of the front-end, you can run more complex and robust tests by experimenting with the backend data layer:

[perfectpullquote align="full" bordertop="" cite="" color="#16989D"]"An example of that is, pricing so that you can have an e-commerce site and with client-side testing, you could change how it looks, how it feels, what the flows are, the images, all that stuff. But such a huge part of your experience is pricing, and you can't change prices by just changing the number on the front end of your website, you have to change it in the underlying data layer." - Claire Vo[/perfectpullquote]

Optimizely's OKR set is hyper-focused on improving the value customers are getting from their products.

But does all of that autonomy comes at a price? Don't PM's get overwhelmed by how much choice they have to build new products?

It's something Claire actively screens for on her product team:

[perfectpullquote align="full" bordertop="" cite="" color="#16989D"]"I want to screen for the ability to say no because it's so important to know what not to build versus all the million things that you could build." - Claire Vo[/perfectpullquote]

2. World-class product teams are diverse

[perfectpullquote align="full" bordertop="" cite="" color="#16989D"]"Optimizely serves a wide variety of customers and end users, and I want my product team to reflect the diversity and experience of those customers. I don't want to hire the same person over and over again." - Claire Vo[/perfectpullquote]

If you have a rigorous hiring framework, it can result in you building a team full of the same people. It means you'll often have a very singular approach to ideas and problem solving from your team.

If you want to build a more diverse group of people, you have to be clear on the things you won't budge on, and those you will.

Claire looks for entrepreneurial spirit, intellectual honesty and culture ad. These are things she won't budge on:

[perfectpullquote align="full" bordertop="" cite="" color="#16989D"]"I don't budge on people having a strong sense of ownership, I don't budge on people having data analysis capabilities, and I don't budge on intellectual honesty. " - Claire Vo[/perfectpullquote]

What she does budge on is:

[perfectpullquote align="full" bordertop="" cite="" color="#16989D"]"What I do budge on is, is this person more technical or are they more creative? What I do budge on is, do they have a wildly different background than someone else? Do they have a consumer background versus a B2B background? " - Claire Vo[/perfectpullquote]

What is intellectual honesty?

Intellectual honesty helps you think about the solution to a problem from multiple viewpoints, not just your own.

[perfectpullquote align="full" bordertop="" cite="" color="#16989D"]"So I want somebody that can come in and say I'm making these assumptions and I realize from a product perspective, here are the pros and cons, from a business perspective, here are the pros and cons, from a organization and people perspective, particularly when you're hiring in people managers, this is the pros and cons.

And be able to come forward with all those things and have a conversation about what the right decision is, as opposed to presenting sort of a single sided point of view that hides sort of the drawbacks or unseen consequences of that decision." - Claire Vo[/perfectpullquote]

Given the PM's in Claire's teams are wearing many hats, e.g., what's right for the customer, the business, having intellectual honesty means they're honest about the solution they've come up with, the potential pro's and cons and potentially if there are unknown problems they don't have great answers for.

3. World-class product teams have experimentation as part of their culture

Like Pinterest and SurveyMonkey, no individual or team own experimentation, instead, it's part of every PM's job.

Every Wednesday morning Optimizely have an experiment review meeting, where anyone across the organization can recommend experiments they want to see run.

Each experiment idea must be in a specific format with a lot of focus on the hypothesis:

[perfectpullquote align="full" bordertop="" cite="" color="#16989D"]"We spend a lot of time inspecting hypothesis. Is it crafted in a format that we believe is measurable, I like the format of, if we do X, we expect to see Y as measured by Z. So we ask for every idea that comes through the process, do you have a sound hypothesis? " - Claire Vo[/perfectpullquote]

The experiments are focused on user behavior that drive metrics listed in the OKR model. Each experiment should list what the potential ROI is if successful.

Along with the hypothesis and potential ROI, Claire puts a lot of focus on consistency of experimentation, she wants her team to get into a regular cadence of running experiments.

[perfectpullquote align="full" bordertop="" cite="" color="#16989D"]"I would much rather have a team that runs two experiments every month, than a team that runs 20 experiments in one month and never runs an experiment for the rest of the year." - Claire Vo[/perfectpullquote]

You can reach out to Claire on Twitter and LinkedIn.

The podcast provides a more in-depth look at these topics, so if you enjoyed reading the above, please do give it a listen.

And until next time,

Happy Growing!