Cost for QD employees to rent an apartment in Chico, CA. $1,200/month. Please turn off your ad blocker in Quality Digest
Our landlords thank you.
Tom Pyzdek
Published: Wednesday, January 11, 2012 - 14:20 One of the cornerstones of quality and lean Six Sigma is data: “We insist on it.” “Don’t tell us what you think the situation is; let the data do the talking.” “In God we trust—all others bring data.” You get the idea. An unfortunate side effect of this emphasis is the proliferation of useless data. If the useless data weren’t used, then collecting the data would merely be a waste of time. But if a person’s performance is being measured by these data, you can bet your last euro that the measurements will get a lot of attention, and it will drive a lot of behavior. And if the system doesn’t change, there’s still one way to make the measurements look better: cheat. I often open my face-to-face training sessions with Dr. Deming’s Red Bead Experiment. It’s a great icebreaker, and it introduces some important statistical ideas. The experiment is actually a game with very simple rules. “Willing workers” are required to use a paddle with holes in it to sample beads from a container that has red and white beads in it. “We don’t want any red beads,” the workers are told. To drive the point home, there are “quality Inspectors” to check the samples for the unwanted red beads and to record the results, and “supervisors” to use the results to “coach” and discipline the hapless willing workers. Before the game concludes, there are always participants who, seeing a bunch of red beads on their paddle, quickly dump the sample back before the count can be made. Others deliberately pick out red beads and throw them back. Still others bring partially filled paddles to the quality inspectors. There are all manner of ways to try and beat the system. And this is just a fun game, played for no stakes at all. Imagine what people do when real consequences are on the line, such as pay and promotions. The most serious games are probably played in totalitarian countries, where factory managers are measured and sometimes executed when the results are less than required by the authorities. According to the History Learning Site, in Stalin’s Russia: Factories took to inflating their production figures and the products produced were frequently so poor that they could not be used—even if the factory producing those goods appeared to be meeting its target. The punishment for failure was severe. In the book Eat the Rich (Atlantic Monthly Press, 1999), author P. J. O’Rourke tells us that in the former USSR: The trouble wasn’t that factory managers disobeyed orders. The trouble was that they obeyed them precisely. If a shoe factory was told to produce 1,000 shoes, it produced 1,000 baby shoes because they were the cheapest and easiest to make. If it was told to produce 1,000 men’s shoes, it made them all one size. If it was told to produce 1,000 shoes in a variety for men, women, and children, it produced 998 baby shoes, one pump, and a wingtip. If it was told to produce 3,000 pounds of shoes, it produced one enormous pair of concrete sneakers. Perhaps O’Rourke is exaggerating, but the point is still essentially valid: Metrics can—and probably will—be gamed. In lean Six Sigma there’s a common metric gaming activity that I call Denominator Improvement. One of the most popular metrics is defects per million opportunities, or DPMOs. The formula itself is quite simple: DPMO = 1,000,000 x Defects/Opportunities. If someone’s performance is being measured using DPMOs, he can make the metric look better by reducing defects (the numerator), or by increasing the number of opportunities (the denominator). For example, we might be interested in the number of typing errors in this post. The DPMO metric might be 1,000,000 x Errors/Total Words. But if this number didn’t look good enough, I might also use 1,000,000 x Errors/Total Letters, or 1,000,000 x Errors/Total Characters, counting spaces and punctuation. The solution to metrics gaming is to use metrics to guide improvement, not to measure the performance of people. Metrics should be limited to those numbers that quantify an important outcome (Y metrics), or quantify an input that is critical to the quality of the outcome (a CTQ or X metric). The reason for quantifying these things is to discover, validate, and use a transfer function—such as Y = f(x), a model of the cause-and-effect relationship—to guide improvement planning and activity. When metrics serve a useful purpose such as this, the tendency to manipulate and game them is, if not eliminated, at least reduced. Quality Digest does not charge readers for its content. We believe that industry news is important for you to do your job, and Quality Digest supports businesses of all types. However, someone has to pay for this content. And that’s where advertising comes in. Most people consider ads a nuisance, but they do serve a useful function besides allowing media companies to stay afloat. They keep you aware of new products and services relevant to your industry. All ads in Quality Digest apply directly to products and services that most of our readers need. You won’t see automobile or health supplement ads. So please consider turning off your ad blocker for our site. Thanks, Thomas Pyzdek’s career in business process improvement spans more than 50 years. He is the author more than 50 copyrighted works including The Six Sigma Handbook (McGraw-Hill, 2003). Through the Pyzdek Institute, he provides online certification and training in Six Sigma and Lean.Gaming the Metrics
Use metrics to guide improvement, not measure the performance of people
Our PROMISE: Quality Digest only displays static ads that never overlay or cover up content. They never get in your way. They are there for you to read, or not.
Quality Digest Discuss
About The Author
Tom Pyzdek
© 2022 Quality Digest. Copyright on content held by Quality Digest or by individual authors. Contact Quality Digest for reprint information.
“Quality Digest" is a trademark owned by Quality Circle Institute, Inc.
Comments
Exaggerating
I was NOT exaggerating. I documented FACTS. I learned use of 6 Sigma at Motorola where I worked 1989 to 2002. I presented the FACTS to managers in the quarterly quality presentations. As a safety professional I KNOW facts from fiction
Gaming
I always cite Brian Joiner who wrote in Fourth Generation Management that there are three things you can do to hit your numbers:
1) Improve the system
2) Distort the system
3) Distort the numbers
#2 and #3 are easiest in the short term, sadly. I've seen a lot of it in my career.
Another side effect of the data talk is the unfortunate quote "If you can't measure it, you can't manage it." That's often wrongly attributed to Dr. Deming, when he taught the exact opposite - that some of the most important things in a business can't be measured.
Good Article
You hit the nail on the head, Tom! Are we going to use the data to whack people with, or are we going to use the numbers to help people improve and succeed? Gee, which one stands the better chance of getting us the outcomes we want? (Correct answer is: "Duh!")
I spent more than 30 years of