5 Traits of Best-in-Class Optimization Teams

What are your best customers doing?

That is the #1 question I hear from customers on a day-to-day basis. How do others companies do optimization and testing? It’s a great question.

Based on thousands of interactions with Optimizely customers and four years of enterprise enablement, I can confidently point to five traits that all best-in-class optimization teams possess:

1. They’ve established a habit of optimization.

2. There is a clear “owner” of the optimization program.

3. The C-suite cares about optimization (and acts on it).

4. Optimization goals are aligned with key company metrics.

5. They make it fun.

1. They’ve established a habit of optimization.

“We are what we repeatedly do. Excellence, then, is not an act, but a habit.”

—Aristotle

Today, Aristotle’s adage above still rings true. It also highlights a cornerstones of all successful optimization programs: HABIT.

In Charles Duhigg’s The Power of Habit we learn that habits are a three-step loop: cue, routine, reward. The cue is what triggers the routine. Thankfully, when Google launched Google Calendar in April of 2006 humans obtained an easy way to design their own cues. Enter: the “repeating” meeting for the win.

Sounds trivial, but all of our best customers embrace some form of the recurring meeting format. It is the forcing function that furthers their optimization endeavor.

Do you have a repeating meeting on your calendar to create your company’s optimization habit?

A few other examples of habit-forming meetings:

• Weekly optimization standup (Forbes)—technical review of pre-launch experiments

• Weekly results review (HomeAway.com)—identify learnings from completed tests

• Quarterly KPI evaluation (Crate & Barrel)—goal alignment, deliverables for the quarter

• Weekly prioritization meeting (TicketMaster)—stack rank based on effort vs. impact quadrants

2. There is a clear “owner” of the optimization program.

When it comes to execution, a world-class optimization program relies on people. Humans who work to design, manage, and ultimately execute against a plan.

Whether your team is an army-of-one or 50+ people, the linchpin is most certainly the program manager, e.g. the optimization “owner”.

Ask yourself: Who wakes up in the morning and thinks about optimization at my company? If there isn’t an owner, assign one or hire one. Otherwise your optimization program will likely flatline.

Here is what this role typically looks like on LinkedIn:

This critical role takes the time to:

• Crowd-source testing ideas from the org

• Consolidate them in a testing backlog

• Prioritize the backlog based on KPIs and effort vs. impact

• Communicate with—and get buy-in from—stakeholders

• Green light tests for execution in a centralized project plan (see below)

• Track and communicate results and inferred learnings

• Iterate. Use what was learned to inform the go-forward strategy.

This is a lot of work for someone who isn’t 100% committed. For this reason, they can’t be a part-time lover (yes, that’s a Stevie Wonder reference on an optimization blog).

If you don’t have the resources internally, its not the end of the world. Look to evaluate Solutions Partners who can help steer the ship for you.

Sidenote: Best-in-class programs also have substantial access to developer/IT resources. If you don’t have this benefit, it might be time to make some new friends in that group. Arming yourself with Red Bull, quirky dev humor, and knowledge of the new Civ will earn you major points. Developer support of your optimization program will add substantial octane to the engine. Rev it up!

3. The C-Suite cares about optimization (and acts on it).

If your leadership team cares about A/B testing and optimization you’re in good place. But talk is cheap, so we look for clues that they actually walk the talk. Does your leadership team:

• Allocate strategy & technical resources to optimization?

• Review results regularly?

• Suggest ideas for testing?

• Say, “I don’t know, let’s test it”? or “We should test that.”

• Provide guidance and direction on quarterly optimization goals?

• Prevent certain stakeholders from blocking the deployment of winning tests?

Without executive sponsorship, building a best-in-class optimization program can be a scratch & claw uphill battle. The Roadmap to Building a Testing Culture eBook contains a number of ideas to get their buy-in.

4. Optimization goals are aligned with key company metrics.

In our 6 Best Practices article we highlight “defining quantifiable success metrics” as the #1 driver of success. But the industry leaders take it a step further: their testing goals are not only well-defined, but also aligned with their key company metrics. For example, a retail website like The Honest Company would align their goals as such:

This alignment helps them deprioritize less-relevant tests by keeping their eye of the prize, i.e. improve Customer Lifetime Value, and ensures your testing program doesn’t go off the rails into random-behavior land.

(Disclaimer: I don’t agree with the premise of this cartoon at all, but I do think its hilarious.)

5. They make it fun.

The best-in-class companies make optimization fun.

Three weeks ago I attended the Zappos Culture Camp: a 3-day deep dive into their special sauce that’s fueled their ridiculous growth to $1B+ in revenue and compelled Amazon to acquire them in 2009 for 40x EBITDA. It’s also worth mentioning that Zappos AOV is ~$130 vs. Amazon’s ~$50. Boom goes the dynamite.

Impressive numbers aside, Zappos is unique for another reason: they’ve built a company culture that intentionally values fun, e.g. Zappos Family Core Value #3: Create Fun and a Little Weirdness.

Here are a few ways we’ve seen optimization made fun:

• Submit an idea competition! (IGN)

• Test of the week/month (HomeAway.com)

• Quarterly A/B Headline Hackathon (CNN)

• Company-wide recognition for the person that suggested a winning experiment (A&E)

• Host a quarterly off-site and invite optimization experts to speak (Mozilla)

I hope you’ve enjoyed reading this! If something resonated with you—or I completely missed something—please post a comment below.

Don’t forget the technical side…

The 5 traits above are mostly organizational & non-technical in nature. I’d be remiss to not mention the technical side of the optimization yin-yang. Here are the 3 Technical Best Practices we recommend based on best-in-class optimization programs:

1. They make their product and visitor data available client-side.

• This is a game-changer.

• This could be in the form of cookies, Javascript variables, custom tags, etc.

• This is really important because you—as a technical person—have now enabled non-technical folks to leverage the data that’s available.

2. They really understand all the nooks & crannies of their site.

• What cookies does your company already use? What’s in those cookies?

• How do you leverage the cookie data for optimization and personalization?

• Knowledge of their website(s) page hierarchy and URL structure

• Strong grasp of the moving parts of your site: dynamic content, AJAX, etc.

3. They ensure that existing processes doesn’t stand in the way of velocity from a development perspective.

• Streamlined QA and development process due to the reduction of red tape

• Is it really necessary to create a fully functional design and requirements doc for a CTA or image change?

• Don’t let perfect be the enemy of good.

Thoughts on organizing & measuring a CSM team

Reposted from an interview with Aircall.io on August 8, 2015.

Optimizely is a San Francisco-based startup and the leader in A/B testing and experience optimization. I recently sat down with Luke Diaz, a manager on the Customer Success team—and founder of DBT Ventures—to share his experience and advice for organizing and measuring a customer success team.

Luke currently leads a 15-person Customer Success Manager (CSM) team in charge of over 80% of Optimizely’s revenue. The CSM team manages launch (onboarding), success management (adoption & value), renewals (retention) and expansion (account growth; in tandem with the Sales team).

The CSM team is one team within the larger 70-person Customer Success team at Optimizely which includes Technical Support, Strategy, Solutions Architects, and Education.

From my (personal) standpoint, Optimizely is very advanced on the topic. Yet every startup—whatever their stage of development or price point—can learn from Luke’s very actionable tips:

• Measure the value customers extract from your product

• Start customer success with the sales team

• Transform your customer’s organization to achieve success

In order to effectively lead the CSM team, Optimizely focuses on 3 metrics:

1. Customer value derived from the product

2. Customer satisfaction

3. Revenue generation

Luke articulates below on how these objectives are translated into processes and culture.

Measure the value customers extract from your product

Optimizely tracks the activity of each Enterprise customer: usage logs, number of A/B tests run, etc. Seems obvious – all serious SaaS businesses (should) do that. To do so, Optimizely uses a blend of Totango (a Customer Success Intelligence software) and proprietary regression models built by their data science team, e.g. churn score, upsell score, account potential score, etc.

What’s more, they follow the number of successful experiments (in their case, delivering a clear A/B winner), and, as much as they can they can, actuallycompute the value in $ generated by successful experiments.

As Luke says:

“When I plan a customer business review, I hope to have a very clear, factual view of the $, or millions of $, we’ve helped them generate using Optimizely. If it doesn’t make dollars, it doesn’t make sense.”

ROI is most explicit when displayed in revenue for, say, a Retail or E-commerce site. But this information is not always easy to gather, especially when dealing with SaaS or media businesses. How do you measure the actual value of improving lead conversion on an optimized sign-up form, or additional clicks on a media module?

Optimizely’s team is  considering adding such functionality and reporting  into their product, or integrating with technologies like Moat (ad viewability) to convey ROI better.

Beyond this #1 metric, Optimizely measures customer satisfaction (metric #2) using NPS (Net Promoter Score) surveys.  They run regular NPS surveys at brand level (4x a year), at the end of the onboarding phase (8-10 weeks), and after each interaction with the support team. The NPS results give an idea of the service quality, the perceived value, and a proxy for customer loyalty.

Finally, the #3 metric is revenue. Luke and his team are incentivized on the net retention rate of their portfolio. They’ve actually manage to have a negative net churn rate (reminder: net churn formula = gross churn – expansion + contraction), around (.5%), in recent months. One simple metric used for almost the entire CSM team.

Luke is currently experimenting with two additional functions on the CSM team:

• Launch Manager: a new role dedicated exclusively to the enterprise onboarding process (measured by NPS and volume)

• Mid-Market CSM: higher volume, lower touch account management approach (measured by renewal rates & net retention)

Start customer success with the sales team

As Luke shared with me:

“I feel lucky to work with the Sales team we have at Optimizely. They are some of the most empathetic and intelligent folks I’ve ever worked with, and they put the customer’s needs and goals first and foremost to ensure a proper fit. Sales and Customer Success have crafted a strong partnership which is imperative to achieve best-in-class net retention.”

Beyond the performance of the Customer Success team, the complementary process to ensure a negative churn rate is a sales validation process implemented by Optimizely’s VP of Sales, Travis Bryant: whenever a sales rep identifies a new account, he or she is  required to populate a 45-question validation form before closing the deal. Simple, straightforward, objective, Yes/No questions designed to determine whether  the new prospect is actually a good fit for Optimizely (Steli Efti, from Close.io, shares a similar philosophy although he uses different method).

According to Luke, the validation form by itself isn’t what guarantees the quality of new customers, but it sets the baseline for having a culture of customer success and retention inside the Sales team. It implies a shared agreement between the Sales and Customer Success teams that each customer is a good fit for Optimizely.

In addition, Luke  personally screens every new customer and gives a special importance to validating their needs. According to Luke, with this process, truly “bad” enterprise customers are extremely rare.

It’s common practice in businesses to build retention metrics into Sales people’s incentives to avoid chasing customers without a longer term view. Optimizely’s approach is interesting in the sense that it clearly incentivizes Sales people on revenue generation, but ensures coordination with the Success team—which owns net retention—using a validation process, along with tight collaboration.

In the spirit of transparency, they entire Optimizely organization receives an email alert whenever an enterprise customer decides to churn.

“This simple workflow ensures that all employees—from the intern to the C-level—are in the loop when we fall short for a customer, and it often sparks internal dialogue about priorities and opportunities to change, iterate and refine.”

Transform your customer’s organization to achieve success

Believe it or not, a major part of the customer’s ultimate success does not rely on your customer success team, your sales team, or your product, but rather: in your customer’s organization. We came to this conclusion as Luke was sharing his best and worst customer success experiences.

Worst? The biggest challenge is when the customer lacks the skills or people to execute: generate ideas, setup experiments, QA, measure, rinse & repeat. According to Luke, the main reason for “customer failure” with Optimizely is a disconnect between a buying decision made by a senior executive and the actual resources available in the team to use it and get value out of Optimizely’s software. The help close this gap, Optimizely has curated a network of 80Solutions Partners to help their customer build—or accelerate— their optimization program.

Another challenge: earning executive mindshare at the VP and C-suite level. In a recent survey of 500 CMOs, optimization ranked #12 out of 17 various marketing priorities.

“We are crafting our sales and account management strategy to uplevel the conversation and earn executive sponsorship. This strategy, along with the coming product releases (e.g. personalization) will ultimately make Optimizely unturnoffable.” 

Best? Luke’s most memorable customer success happened when one of  the Customer Success team members managed to convince a client to make a hire in order to lead and improve optimization initiatives, after demonstrating that the first tests generated a 15% increase in revenue. Optimizely provided the data to help the customer’s Marketing Director make the case for net new headcount, thereby creating transformational change in the company. “In my opinion, it was one of our proudest moments,” Luke said.

Worst scenario, best scenario: both are related to the customers’ resource allocation, and that’s one key lesson for all SaaS out there: the ROI your customers derive from your software is correlated to the resources devoted to it. Convince your customers to organize for success!

http://blog.aircall.io/customer-success-optimizely/