Is It Time to Bring Data to Managing?

There’s plenty of evidence on how to get the best out of workers, but businesses often ignore it

Published: Thursday, August 30, 2018 - 11:03

Alan Colquitt is a student of the ways people act in the workplace. In a corporate career that spanned more than 30 years, the industrial-organizational psychologist advised senior managers and human resources departments about how to manage talent—always striving to “fight the good fight,” he says, and applying scientific rigor to his job.

Should executives ask employees for hiring referrals? Colquitt would consult the research to see if that would bring in better candidates. How to get more women into senior management? Colquitt would dig into studies that revealed the reasons for the stubborn endurance of the glass ceiling.

And then he hit a ceiling of his own.

A Fortune 500 firm where he worked had put in place a compensation system that was making employees miserable. Colquitt hadn’t been the one who implemented the system, which gave better raises and bonuses to those who scored high on a five-point performance scale. But people complained to him about it, incessantly. He decided to push upper management for change.

True to his roots, Colquitt reviewed the published literature and combed through internal data to show higher-ups where, exactly, things were going wrong. The evidence led him to a stark conclusion: The firm’s performance assessments and pay structure were completely counterproductive, reducing happiness of individual workers and hurting the enterprise as whole.

Colquitt recommended that his employer scrap the system. The company’s CEO backed him, but many others in the organization, including heads of the human resources and compensation departments, pushed back hard on a total revamp.

Colquitt kept arguing. No dice. After a couple of years, exhausted and ready for a career shift anyway, he gave up. He left corporate life and became an affiliate research scientist at the Center for Effective Organizations at the University of Southern California. He began teaching, speaking and writing, and published the book Next Generation Performance Management: The Triumph of Science Over Myth and Superstition (Information Age Publishing, 2017).

“I was pretty outraged by it all,” he says today. “What we do in organizations has very little relationship to what the science says we should do.”

Getting companies to pay attention to science and engage in so-called “evidence-based management” is a challenge that has been driving industrial-organizational psychologists nuts for the better part of 20 years. Whether it’s hiring staff or determining salaries or investing in technology, managers making high-stakes decisions have extensive scholarly literature at their disposal: studies conducted throughout more than a century in labs and in the field, vetted through peer review, that show whether pay incentives drive internal motivation (often not); whether diversity training works (only under the right conditions); whether companies should get rid of performance ratings (yes, Colquitt would say); how to train effective teams; and more.

Executives love hard numbers, and they desperately want to know how to keep their best employees, how to make more widgets, and how to be more creative. So you’d think they’d lap up the research. “It’s hard to find students in graduate school who don’t hear the idea of evidence-based management and say, ’Yes! Of course!’” says Neil Walshe, an organizational psychologist who teaches the approach at the University of San Francisco School of Management.

Except most companies don’t. Occasionally, a firm will make a splash—the poster child these days is Google, which gets kudos for its data-centric, research-based “People Operations” (aka human resources) department. But most executives would rather just copy another company’s proven ideas than do the hard work of assessing evidence relevant to their own circumstances. Managers falter, victims of inertia (“but we’ve always done things this way!”) confusion (“industrial-organizational what?”), even downright hostility to expertise.

Interest in evidence-based practices may get a boost as more and more companies start delving into data analytics the way Google has, observing their own operations and putting the information to use in thoughtful ways. Perhaps, proponents hope, managers who open their minds to analytics will also open their minds to other new ways of thinking, seeing the value in evidence.

Or maybe human nature will keep getting in the way.

“There are really good reasons why people don’t use evidence, and changing that is hard,” Colquitt says.

The science of business

Science-inspired ideas have been applied to business since at least 1911, when mechanical engineer-turned-management consultant Frederick Winslow Taylor’s Principles of Scientific Management (Harper & Brothers, 1911) applied insights from engineering to improve efficiency, arguing that “the best management is a true science, resting upon clearly defined laws [and] rules.”

Taylor worked with Bethlehem Steel to optimize the volume of pig iron a worker could load onto railroad cars in a single day. He studied “the tiring effect of heavy labor” and tasked a young assistant to look up “all that had been written on the subject in English, German, and French.” He conducted experiments to figure out how much iron a man could consistently haul and through a process of analysis determined that a “first-class man,” with the right strength and pacing, should be able to manage 47 tons. He urged managers to move workers who couldn’t handle such a load into other roles.

In later decades, researchers studied industrial behavior with ever-increasing rigor, the field of industrial-organizational psychology was born, and academic work increasingly informed business practices, within human resources and without. During World War I, the military used assessments to place soldiers in jobs where they’d be most successful. In the 1920s and 1930s, a series of famous studies at Western Electric’s Hawthorne plant in Cicero, Illinois, influenced managers to pay attention to social interactions on teams.

Japan’s postwar economic boom was also built on research, including the 1950s-era innovations of statistician W. Edwards Deming, who focused on product quality, among other things, as a driver of business success. It didn’t take long for American companies to adopt “total quality management,” as the trend became known in the United States, when Japanese firms began to threaten American predominance.

With the start of the 21st century, another concept was percolating up through the industrial-organizational psychology ranks: an approach called “evidence-based management,” articulated and championed by a Carnegie Mellon professor named Denise Rousseau. She gave an impassioned speech on the subject in 2005, at the annual meeting of the Academy of Management in Honolulu.

Rousseau had always assumed that companies paid attention to the research she and her colleagues so carefully produced, but slowly it began to dawn on her that that wasn’t the case. It was an epiphany that “blew my mind,” she says today. Managers rejected scientifically proven strategies and refused to abandon practices the literature didn’t support: things like paying executives outlandishly more than rank-and-file employees. Bosses made decisions based on gut feeling. They copied blue-chip companies like GE and Coca-Cola, even when what those outfits did had little relevance. They chased trends.

Rousseau and other industrial-organizational psychologists thought this seemed like a colossal waste of time, effort, and money. They saw a model for change in a movement called evidence-based medicine, which urges physicians to consider the best available external evidence when deciding how to treat patients. Increasingly, beginning in the 1990s, doctors were expected to lean on research, not just go with their guts.

Shouldn’t businesspeople, similarly, take stock of the work that psychologists had so carefully produced?

“It’s time to start an evidence-based movement in the ranks of managers,” exhort Stanford Business School professors Jeffrey Pfeffer and Robert I. Sutton in “Evidence-Based Management” (Harvard Business Review, January 2006). The pair tartly opine that if doctors practiced medicine the way managers practiced management, the morgues would be packed and the courts brimming over with malpractice lawsuits. Managers should “relentlessly seek new knowledge and insight” to hone key practices, the scholars say.

The idea took root. Researchers like Rousseau wanted companies to read their papers, to be sure—but they also urged critical thinking in a broader sense, in which everything from science to internal surveys to gut feelings is considered in a systematic way, following a six-step process. The concerns of evidence-based managers are “Why are we doing this? What is the problem we’re trying to solve? How do we know the solution will solve the problem?” says Eric Barends, managing director of the nonprofit Center for Evidence-Based Management, an international network of experts.

Worried about revealing trade secrets, companies are loath to talk openly about their real-world experiences with evidence-based management. Still, the approach can yield results.

Take the case of one company that couldn’t retain software engineers. It asked Cheryl Paullin, a Minneapolis-based industrial-organizational psychologist who heads the talent management and analytics division of the Human Resources Research Organization, a nonprofit that advises HR departments, if it should raise salaries to keep programmers on board.

Reviewing the academic literature and a variety of metrics within and outside of the company, Paullin and her colleagues determined that programmers were leaving not because of pay but because they weren’t getting the training they wanted. “We were able to say: ‘Don’t try to keep them by focusing on their pay—that’s not what’s causing the problem,’” Paullin says.

In another case, documented by Barends for a forthcoming evidence-based management textbook, Ctrip, the largest travel agency in China, conducted a randomized trial to help determine if allowing call center employees to work from home would improve their individual performance (several academic studies suggested it would). The company chose 250 employees for the three-month experiment, assigning those with even-numbered birthdays to work at home and those with odd-numbered birthdays to work in the office. Remote workers increased their performance by 13.5 percent over their colleagues in the office and used fewer sick days, too. “Stunned” by the result, Ctrip’s CEO decided to adopt remote working for all call center employees.

Or there’s the far from straightforward problem of assigning salary ranges to different types of workers. To evaluate jobs and set pay, many companies still rely on outdated systems designed in the 1940s that assign higher salaries to people who are managers or in charge of budgets and give short shrift to newer sorts of jobs that are very valuable to 21st century firms—roles like project management or functions requiring expert skills and knowledge, says Philipp Schuch, a cofounder of Gradar.com, a Dusseldorf, Germany-based HR tech startup.

So Gradar is using an evidence-based approach to build a web-based job evaluation tool. Months were spent studying existing systems to understand what criteria was used to grade jobs and derive pay scales. Then a comprehensive literature search was conducted to come up with updated, requirements-based criteria that make more sense for today’s workplace: things like responsibility for key functions and projects, and not just people and organizational responsibility.

The company built a pilot system, then tested and retested it over and over to validate its results against established systems and other jobs-related data. Then an online system was built and tested and retested again (verification is a crucial part of an evidence-based approach). Today, more than five years after Schuch and colleagues began thinking about Gradar, the company is working with 100 medium to large companies around the world.

They range from auto parts manufacturers to theater companies to universities—“and it still works across all the different jobs. We get consistent results,” says Gradar cofounder Ralf Kuklik.

Why managers won’t commit

Schuch and Kuklik are believers in their tool—but they’re also realists. Schuch worked for years as head of compensation and benefits at large German companies and he’s been paying attention to what other companies do with evidence-based management.

“It’s not much, honestly,” he says.

That’s a common refrain.

“We’d love to see a commitment from a leader that says, ‘I expect our decisions about people and work and the organization to have evidence behind them,’” says John Boudreau, research director at the Center for Effective Organizations, housed in the University of Southern California’s Marshall School of Business. “I don’t know that I have seen examples of that. Especially at the high level, the CEO level.”

“I’m a little baffled that it’s not more widespread,” says Jennifer Kurkoski, director of Google’s People Innovation Lab (PiLab), the internal research and development team behind the company’s People Operations department. “Companies spend billions on R&D, almost none of which is devoted to making people work better. It’s not something we understand yet. And we should.”

But there are many reasons why managers have been slow to embrace evidence-based management.

It’s a lot of work. Companies must spend a great deal of time, effort and money to assimilate research findings, or to test and validate new policies or systems. “Most people want to put things in place quickly, and get it done,” says Elaine Pulakos, president of PDRI, a Washington, DC-based talent management company. Executives, always with an eye on the bottom line and the next quarter’s results, often see this sort of research as overhead they can’t afford.

People fear change and risk. Even though an evidence-based management approach may ultimately yield better results, the perceived safer route is hewing to well-known “best practices” championed by other companies, and promoted by consultants who may or may not have done rigorous study. “People get enamored with something they can easily implement that someone else has tried before them,” says Pulakos. If Exxon Mobil or Google has scored with some initiative, she adds, “it makes it safe.” But maybe irrelevant, too.

Managers put more faith in intuition than they put in science. “We’re all experts on human behavior, right?” jokes organizational psychologist Ed Lawler, of University of Southern California’s Center for Effective Organizations. It’s an abiding sense that’s often flawed: Sometimes, industrial-organizational psychology research reveals that algorithms are better than people at particular tasks, such as initial screenings for new hires. But “people tend not to like findings that don’t present humans in a good light,” says Sara Rynes, an industrial-organizational psychologist at the Tippie College of Business at the University of Iowa.

Parsing the scientific literature can be hard. Managers, unlike doctors, aren’t required to have any kind of advanced training, and often can’t read a scholarly report or engage in the sort of statistical analysis needed to understand internal employee data. At the same time, academics catch fire for not making their findings more readable, or for publishing their work in prestigious journals that keep studies hidden away behind paywalls—pushing managers toward popular business books and articles that do an uneven job of presenting research correctly.

“It’s hard to find the research, and it’s hard to read, and it’s hard to interpret,” Colquitt says. “There are so many more channels to get information... it’s hard for leaders or HR professionals to sort the wheat from the chaff.”

Making matters worse—ironically—the very people who champion the science-based approach haven’t yet proved that it works with the kind of rigorous study that they would like. In that sense, “the evidence for evidence-based management is almost nonexistent,” admits Rob Briner, an industrial-organizational psychologist and scientific director of the Center for Evidence-Based Management.

In a paper published in the Annual Review of Organizational Psychology and Organizational Behavior in 2017, Rynes and co-author Jean Bartunek of Boston College examined 134 scholarly articles about evidence-based management. Most were essays and other pieces advocating or criticizing the approach, talking about how to teach it, and the like. Only about a fifth were empirical studies reporting research or reviewing such studies. The authors highlighted just a handful of those as “exemplary”—noting that many focused on small numbers of subjects and relied on self-reporting from managers for data, a method “known to be fraught with numerous biases and opportunities for error.”

“People want more evidence that when people use our studies it actually does something,” says Rynes.

As Google goes . . .

Advocates for evidence-based management think their approach may start looking more interesting to more people now that companies are embracing big data analytics: slicing and dicing truckloads of behavioral information, much of it collected through internal workplace computer systems, to dig up insights. Some of this information sits in databases, other bits are embedded in operational systems, and can be mined.

Here, Google reigns supreme. It’s in the business of collecting and analyzing information, after all. Kurkoski’s team, heavy with doctorate degrees, questions all kinds of assumptions about organizations. Then it consults the research, tries to find data within its own operations to shed light on the question, and tests new ways to solve problems. Questions like: “Do managers matter?” (yes, because the best ones boost job satisfaction among workers); “Why are women leaving our company?” (industry-standard, 12-week maternity and paternity leaves are too short); even “What shape of lunch table will get co-workers talking?” (a long one).

Kurkoski is close-lipped about a lot of what Google does—she won’t share how many people are on PiLab’s staff, for instance—but the company has earned a lot of attention for its work in the business and popular press. A 2016 article in The New York Times Magazine, for instance, detailed a 2012 initiative known as “Project Aristotle,” designed to figure out what made effective teams work and what made bad ones fizzle. The company ultimately homed in on “psychological safety”—how comfortable workers feel taking risks, a well-studied subject in the organizational psychology canon.

The brilliance of Google’s approach was the way it used science to encourage workers to talk about their feelings, one Google manager who went on to apply the findings told The Times. “By putting things like empathy and sensitivity into charts and data reports, it makes them easier to talk about,” he said.

Colquitt is among those who think the new rage for data analytics might spark renewed interest in evidence-based management—the operative word being “might.” He pounds out blog posts, stuffed with research citations, when the NFL decides to fine players who don’t stand for the national anthem, or when United Airlines toys with converting its bonus system into a winner-takes-all-lottery. He’s digging deeper into the problem of performance management and pay.

And the fodder keeps coming. Studies that find open offices don’t, in fact, encourage conversation and collaboration. Studies that find employees resent the corporate fad of hot-desking—jumping from desk to desk instead of having a dedicated workspace, based on a notion that this will spark synergies and blue-sky thinking.

In one recent paper calling on industrial-organizational psychologists to put “an end to bad talent management,” Colquitt and his co-authors called out companies who fall for consultants promising to help them understand “the brain science of millennials” and other trendy topics, with little or no evidence for any of it.

“We needed to write about it and put these things to bed,” Colquitt says. He adds, as if a dark cloud is momentarily passing overhead: “But no one reads these papers anyway—so they won’t stay in bed long.”

Then it’s back to the talks and the blogs and the books, and fighting that good fight.

First published July 26, 2018, in Knowable Magazine.

About The Authors

Eryn Brown

Eryn Brown is a freelance writer and editor. Her work has appeared in the Los Angeles Times, The New York Times, Nature, and other publications. She graduated from Harvard with degrees in history and literature, and was a writer at Fortune magazine in New York for seven years before moving to Los Angeles.

Knowable Magazine

Knowable Magazine is an independent journalistic endeavor from Annual Reviews, a nonprofit publisher dedicated to synthesizing and integrating knowledge for the progress of science and the benefit of society. Sign up for Knowable Magazine’s newsletter.