Zillow – Steer Clear Of The Blind Spots That Derail Experiments
by Danielle Kost
Should companies let employees keep working remotely after the COVID-19 pandemic ends? Assessing the impact of remote work has involved a lot of guesswork for business leaders, but it doesn’t have to be that way.
Consider the Chinese travel company Ctrip, now known as Trip.com. Its executives ran an experiment almost a decade ago in which the company allowed a randomly selected subset of employees to work from home for nine months while the rest of the team stayed in the office. As it turned out, the employees who worked from home became much more productive. This effort gave the company a critical data point and a unique leg up in thinking about the issue.
CTrip is one of many companies using experiments to guide decisions, according to Harvard Business School professor Michael Luca.
“Executives need to understand when and how to bring data and experiments into their organizations,” says Luca, the Lee J. Styslinger III Associate Professor of Business Administration. “They can’t rely on data scientists alone.”
Experiments have come to have an outsize influence within tech companies from Uber to Zillow, which test everything from new products to algorithms to site features to see what resonates with online users. Google and Amazon are among a growing list of companies running tens of thousands of tests per year. A growing number of companies are applying these techniques to marketing, strategy, and even compensation questions.
However, the use of experiments remains spotty, with even data-driven companies still figuring out how to make the most of them. After observing several technology companies making the same predictable set of strategic mistakes, Luca decided to look at how companies could more effectively leverage structured, randomized experiments to inform decision-making. His goal: Enable C-suite leaders to avoid the blind spots and blunders that can occur when making decisions based on data and large-scale tests.
The project resulted in a new class about experiments for second-year M(BA) students and in his 2020 book, The Power of Experiments: Decision-Making in a Data-Driven World, co-written with Max H. Bazerman, the Jesse Isidor Straus Professor of Business Administration at HBS.
Five experimentation mistakes to avoid
Thoughtful testing can give managers valuable insights, but there are potential pitfalls, Luca says. They include:
Failing to tie experiments with managerial decision-making. Some organizations tend to relegate testing to specialized teams without deep managerial involvement, building silos that ultimately prevent experimentation from permeating the company’s culture. Instead, organizations might learn from the travel site Booking.com, whose leaders have forged a rigorous testing culture that has guided the company to growth, Luca says.
The company has empowered 1,500 employees with varying backgrounds and roles to run tests in virtually every part of the business. Booking.com Chairwoman Gillian Tans has said that the nimbleness that comes with a testing culture helps companies engage employees and develop better products.
Setting too narrow an objective. Managers can create major blind spots when they focus on overly narrow goals. Consider the experience of Airbnb, which ran experiments to evaluate booking strategies but ignored the potential for discrimination. After research by Luca and colleagues found evidence of widespread discrimination on the platform and proposed a path forward for the company, Airbnb made a series of changes and created a process for analyzing bias. Had the executives considered how to account for unintended consequences in their outcome metrics—and perhaps engaged a diverse user group to understand their experiences—they might have spotted this problem sooner, an oversight with significant reputational and legal costs.
Aiming for too short a time horizon. Managers sometimes need to run experiments for weeks or months to fully understand both their short- and long-term results, Luca says. That was the experience of 24 Hour Fitness. A few years ago, the gym chain enlisted the University of Pennsylvania’s Behavior Change for Good Initiative to test new ways to nudge members to exercise. Researchers found that while all of their interventions yielded an initial surge of activity, many of those gains waned with time. Tracking longer-term metrics allowed them to identify which interventions had staying power.
Focusing too much on “what” instead of “why.” Companies often set out to test a particular question or compare a short slate of options with the goal of finding what works best with customers. However, too many stop when they have an answer, rather than ask more questions about the outcome. Alibaba, for example, set out a few years ago to test whether coupons nudged shoppers to buy items they had left in their digital carts. Executives found that these incentives didn’t boost sales, but rather than find out what tactics might actually move the needle, the business moved on.
Not aligning experiments to broader organizational goals. Failing to understand the end goal can lead to misguided decisions about when and how to run such tests. Experiments should answer clear questions that bring an organization closer to its objectives. Does the company ultimately hope to show the value of a product to investors? Is it trying to decide which product to introduce first to make the biggest splash with customers? Would offering a bonus differently boost employee output during a challenging part of the production cycle?
New opportunities for experimentation
While much of Luca’s work focuses on the tech sector, he sees opportunities for valuable tests in a variety of business settings. Here are just a couple of the many questions that experiments can help answer:
How do employees value flexible schedules relative to higher pay? As companies reconsider their post-pandemic working norms, many leaders will weigh the pros and cons of letting employees keep the schedule freedom that comes with virtual work. But, reducing structure comes with its own risks, so companies will likely want a full picture of the potential benefits of such a move to employees and the organization.
A 2016 study by Princeton’s Alex Mas and Harvard University’s Amanda Pallais offers some interesting insights. They found that the average applicant to a job at a large call center was willing to take a significant pay cut to get a predictable schedule. However, two-thirds of applicants weren’t willing to forego any pay for more control over their hours.
While the results of this experiment and CTrip’s experience can be valuable, companies can also test policy proposals on a subset of employees—offering bonuses to those who volunteer to be on call, for example—to see if it’s worth adopting, Luca says.
How do higher wages affect productivity? Many companies pay above-market wages based on the idea that employees will work harder if they’re making more money. However, studies that have probed that assumption have yielded mixed findings. One paper found that an unconditional bonus before a project’s start boosted productivity, while another study concluded that those benefits were short-lived.
A study by Luca, former HBS doctoral student Duncan Gilchrist, and Deepak Malhotra, the Eli Goldston Professor of Business Administration, found that two groups of contract employees—paid $3 or $4 an hour—delivered the same output in a data-entry task. However, a third group paid $3 an hour plus an unexpected $1 wage increase were 20 percent more productive.
An experiment (pdf) by researchers at Harvard, the University of Chicago, and the University of California, San Diego, also found that how employers frame bonuses played a big role in their effectiveness. In the study, teachers who received bonuses at the beginning of the school year—money they had to return if they didn’t meet achievement goals—significantly outperformed educators who received the same bonuses at the end of the year.
Companies can run similar experiments to refine their compensation strategies. They might experiment with pay across different locations and track the impact of productivity of employees. They might change how they communicate about pay (as in the teacher study), or test whether highlighting the reason for a bonus improves output.
An essential skill for executives
As Luca and Bazerman note in The Power of Experiments, companies are still figuring out how to create a culture of experimentation and build a better foundational understanding of when and how to leverage these efforts.
Executives too often rely on intuition, imperfect assumptions, or their own experiences to make decisions without scrutinizing the data, Luca says. While data has gained prominence in many organizations, numbers can only take a business so far if top-level executives lack the tools to act on them.
In his course and book, Luca aims to instill a scientific mentality in students so they can confidently apply these approaches throughout their careers. After all, a deep understanding of experimentation, both the design and analysis, are key skills in today’s data-centric economy and an important source of competitive advantage for companies.
“Business leaders need both frameworks and sound judgment to effectively leverage data and experiments to guide their biggest decisions,” Luca says.