I recently closed the doors of my own consulting company on the prairie in Minnesota and headed back into the wild, wacky, wonderful world of larger consulting groups, joining a group in Northern Virginia. One of the consequences of that transition was that I was unable to meet a couple of speaking engagements, so I promised to write something about dashboards for the people who had planned to attend. This dashboard issue is one of the more egregious problems I see regularly.
Although I can’t really put that whole presentation in writing, I can at least discuss the problem. The presentation was actually a half-day workshop that included W. Edwards Deming’s Red Bead experiment as an opening, followed by a discussion and brief overview of statistical process control (SPC). I see so few people using SPC anymore; most of the companies I work with measure very little and use their data badly. I wanted to include the Red Bead experiment because I consider it a prerequisite, the price of entry. If you understand the red bead, you’d probably never use an idiot-light dashboard.
I suppose it dates me to call these red-green dashboards “idiot lights.” OK, no supposing—it dates me. For you youngsters out there: Back in the old days, one of the options when you purchased a car was a choice of either actual instruments to measure water temperature and oil pressure, or lights that would come on when they were dangerously high or low. In those days, we actually worked on our own cars. We changed the oil, tuned it up every few thousand miles, and flushed the radiator twice a year. Your toolbox was incomplete if you didn’t have a gap gauge (for setting points and plugs), some chalk, and a timing light. There was a chrome-and-plastic tire pressure gauge in almost every glove box.
In those days, knowing the thresholds for your oil pressure and engine temperature were important, so being able to see when they were trending toward the danger zone was a good thing, a necessary thing. Mechanical engineers like my dad and many shade-tree mechanics sneered when Detroit started replacing gauges with indicator lights—hence the term “idiot lights.” I remember car salesmen asking my father if he wanted gauges or idiot lights when he ordered a new car in 1965.
I had the same reaction when I saw an operational metrics dashboard a few years ago that had the “stoplight” indicator. It was hard for me to believe that any manager would decide to ditch all the decision power and knowledge that SPC brings in favor of a colored dot that would just indicate hitting or not hitting a target.
One specific situation sparked my would-be presentation. I was consulting with a large company, working with one of their business units in the role of a Six Sigma Master Black Belt. As a result, I was invited to many management meetings. One regular Thursday meeting concerned a metric that management cared about deeply. Each week, one manager who “tracked” this metric would show the dashboard, then present some conjecture on why the metric was either above the target or below it. If they were above the target, the director would congratulate everyone; thank them for all their hard work. If they were below, the director would ask where everyone thought the problem might be. Usual red-green stuff. The dashboard essentially looked like this:
As it happened, most of these same people also came to a Friday meeting that I had some responsibility for. I had asked for the historical data, and so each Friday, I presented an individuals and moving range chart of these data.
You could even do a capability study with these data, using the target of 28 as a lower specification limit:
Now, anyone who understands the lesson of the red bead knows—with just a casual glance at the process behavior chart—that this process will produce many weeks above the target. They also know, though, that it’s going to produce even more weeks below the target. The metric has been essentially stable for two years, with the centerline just about straddling the target (but just below). The capability study predicts that they’ll be below target about 65 weeks out of 100. The one assignable cause in the moving range chart was due to an exceptionally low week during the Christmas holidays.
So, every Thursday, we’d see the red or green, and we’d pass out wrist-slaps or T-shirts. During red weeks, there would be some furious activity to find out what happened, and take some swift action to tamper with the system. During green weeks, everyone rested on their laurels. So, at least during green weeks, no one was trying to make things worse!
Then, the next day each week, I’d present the latest process behavior chart and capability study. I’d point out that the company was still in statistical control, and that the average was still below the target line. I’d show the capability study and explain that the numbers there predicted that in any given year the company could expect to be below target more than six weeks out of every 10. I’d tell them that things were going to stay this way until someone decided to actually work on the system of causes that was driving the metric. I was usually met by polite silence, with the director eventually saying, “Thanks for that, Rip... anyone have anything else?”
Finally, one brave soul spoke up and said, “Wait a minute. Are you saying that all this stuff we’ve been doing hasn’t helped? We haven’t improved at all?” That opened the floodgates. By the end of that meeting, management had decided to charter a project to look at the overall system and learn what would have to change to move the lower control limit above the target. I was invited to present the control chart each week at the metric meeting (and to the director’s vice president).
I wish I could tell you what happened from that; shortly after that, I was transferred to another business unit, and then the organization ran out of consulting budget. I did hear from the woman who had been tracking the dashboard (and had taken over the control chart), and she said they had at least moved the mean above target, but she didn’t send me the updated chart.
What do we take away from this?
We do pull beads out of a bin. I have presented the red bead experiment hundreds of times during the past 20 years. It almost never fails that someone will say, “Yeah, that was cute, but I don’t pull beads out of a bin for a living. This doesn’t apply to me.” It’s a little like the Oakland scout telling Billy Beane that he should be relying on the experience and gut of his scouts rather than “Pete’s” (Paul de Podesta’s) statistics.
Of course most of us don’t literally pull beads out of a bin. What we actually do can usually be measured, though, so we produce numbers as well as widgets or services. The processes that produce both widget and number are the result of a system of causes. Some of the causes are related to people, some to machinery or technology, some to policy or environment, some to incoming materials—the list goes on and on. Some of these factors can be economically controlled, and some can’t. Until a manager understands the system as a whole and begins to understand all the drivers and interactions in that system, it might as well be pulling beads from a bin because in that case, management has no more control over the results than the foreman in the red bead experiment.
Idiot lights don’t help. Especially when the threshold is arbitrary. The light blinking red and green drives tampering when it’s red, and ignorance when it’s green. There is often (as there was in this case) confusion about what the target even means. When the manager is told to “hit the number,” does that mean hit it every week? That means we need to get the process to a point where the target is outside the control limits. Does it mean hit it on average? Then we need to shift the mean to one side of the target or another. Once we can show that the process is stable, with the mean on the “good” side of the target, we’ve “hit” the target. Just knowing that I’m above or below target doesn’t help me, though, if I don’t know the capability of my process—and I can’t know that from the idiot light.
I did say, “Especially when the threshold is arbitrary.” In the case of my director, as in many other cases, the target was arbitrary and was set on someone’s idea of how much improvement this particular director’s group could be driven to achieve in the fiscal year. The sad fact is that, had she “hit” the target in her yearly expectations, the target would no doubt have been adjusted upward by some other arbitrarily guessed amount for the next year, without the benefit of anyone in leadership knowing any more about the process than they did the last time they winged a target into her expectations.
Myron Tribus, former director of the Center for Advanced Engineering Study at MIT, described this practice particularly well: “... In short, [most leaders’] idea of a good manager is one who sets up a system, directs the work through subordinates, and by making crisp and unambiguous assignments develops a basis to set standards of performance for... employees. [The leader] sets goals and production targets [and] rates employee performance objectively against those targets.” This sounds great, but Tribus’ point is that without the benefit of understanding systems and variation, the targets will be wrong, the rating system will be useless, and the manager’s decisions and actions will produce more harm than good. Tribus was also known for his “perversity principle”: “If you try to improve the performance of a system of people, machines, and procedures by setting numerical goals for the improvement of individual parts of the system, the system will defeat your efforts, and you will pay a price where you least expect it.”
The 85-percent (96%) rule still applies. Deming used to say that 85 percent of the performance of any system belongs to management. During his last few years, he often joked that he’d underestimated that number all along; it should have been 96 percent. His point was that most of the performance of any system is driven by a lot of factors and interactions between them, but most of the management action that he saw was directed at the people working in the system. Managers who ignore the interaction of forces and variation are thus going to be wrong about 85 percent of the time. Heero Haqueboord (president of Decision Partners Ltd.) once joked that we could improve most companies’ performance by 35 percent, if we could just get all their managers to flip a coin when they have to make a decision.
Your world is not red or green, and your dashboards shouldn’t be, either. The purpose of measuring important aspects of a process is to be able to predict how the process is likely to perform, and to be able to make decisions on what to do about that performance. The real world doesn’t give us red or green any more than it gives us black or white. It gives us a whole spectrum in between; although we might have acceptable and unacceptable thresholds somewhere along that spectrum, managing to that threshold requires that we understand where we are in the spectrum—and how wide the spectrum is. Control charts are a great tool for understanding those things.
Is it more difficult? Sure, it’s at least a little harder. Any monkey can be trained to do a flip when the light comes on; I just believe that most managers and executives are a lot smarter than most monkeys, and more should be expected of them. I’m always a little distressed by people (who should know better) who tell me that control charts and SPC are just “too complex” for most managers. Most managers (and most line workers) I’ve met have more than enough intelligence to understand the basics of variation and can readily be taught to use control charts—yes, even to use them well. So let’s start giving managers and executives the actual gauges on their dashboards that will let them understand performance and improve systems, instead of a collection of idiot lights that they can react to.