Featured Product
This Week in Quality Digest Live
Six Sigma Features
Jody Muelaner
A standard set of graphical methods for improving quality
Dirk Dusharme @ Quality Digest
Making better use of taxpayer dollars
Ryan E. Day
Case studies in saving taxpayer dollars with lean Six Sigma
Taran March @ Quality Digest
Lean Six Sigma has arrived on a few campuses. Will more follow?
Ryan E. Day
Take this self-assessment to help determine the lean status of your organization

More Features

Six Sigma News
Floor symbols and decals create a SMART floor environment, adding visual organization to any environment
A guide for practitioners and managers
Making lean Six Sigma easier and adaptable to current workplaces
Gain visibility into real-time quality data to improve manufacturing process efficiency, quality, and profits
Makes it faster and easier to find and return tools to their proper places
Version 3.1 increases flexibility and ease of use with expanded data formatting features
Provides accurate visual representations of the plan-do-study-act cycle
SQCpack and GAGEpack offer a comprehensive approach to improving product quality and consistency
Customized visual dashboards by Visual Workplace help measure performance

More News

Tom Pyzdek

Six Sigma

Gaming the Metrics

Use metrics to guide improvement, not measure the performance of people

Published: Wednesday, January 11, 2012 - 14:20

One of the cornerstones of quality and lean Six Sigma is data: “We insist on it.” “Don’t tell us what you think the situation is; let the data do the talking.” “In God we trust—all others bring data.” You get the idea.

An unfortunate side effect of this emphasis is the proliferation of useless data. If the useless data weren’t used, then collecting the data would merely be a waste of time. But if a person’s performance is being measured by these data, you can bet your last euro that the measurements will get a lot of attention, and it will drive a lot of behavior. And if the system doesn’t change, there’s still one way to make the measurements look better: cheat.

I often open my face-to-face training sessions with Dr. Deming’s Red Bead Experiment. It’s a great icebreaker, and it introduces some important statistical ideas. The experiment is actually a game with very simple rules. “Willing workers” are required to use a paddle with holes in it to sample beads from a container that has red and white beads in it. “We don’t want any red beads,” the workers are told. To drive the point home, there are “quality Inspectors” to check the samples for the unwanted red beads and to record the results, and “supervisors” to use the results to “coach” and discipline the hapless willing workers.

Before the game concludes, there are always participants who, seeing a bunch of red beads on their paddle, quickly dump the sample back before the count can be made. Others deliberately pick out red beads and throw them back. Still others bring partially filled paddles to the quality inspectors. There are all manner of ways to try and beat the system. And this is just a fun game, played for no stakes at all. Imagine what people do when real consequences are on the line, such as pay and promotions.

The most serious games are probably played in totalitarian countries, where factory managers are measured and sometimes executed when the results are less than required by the authorities. According to the History Learning Site, in Stalin’s Russia:

Factories took to inflating their production figures and the products produced were frequently so poor that they could not be used—even if the factory producing those goods appeared to be meeting its target. The punishment for failure was severe.

In the book Eat the Rich (Atlantic Monthly Press, 1999), author P. J. O’Rourke tells us that in the former USSR:

The trouble wasn’t that factory managers disobeyed orders. The trouble was that they obeyed them precisely. If a shoe factory was told to produce 1,000 shoes, it produced 1,000 baby shoes because they were the cheapest and easiest to make. If it was told to produce 1,000 men’s shoes, it made them all one size. If it was told to produce 1,000 shoes in a variety for men, women, and children, it produced 998 baby shoes, one pump, and a wingtip. If it was told to produce 3,000 pounds of shoes, it produced one enormous pair of concrete sneakers.

Perhaps O’Rourke is exaggerating, but the point is still essentially valid: Metrics can—and probably will—be gamed. In lean Six Sigma there’s a common metric gaming activity that I call Denominator Improvement. One of the most popular metrics is defects per million opportunities, or DPMOs. The formula itself is quite simple: DPMO = 1,000,000 x Defects/Opportunities. If someone’s performance is being measured using DPMOs, he can make the metric look better by reducing defects (the numerator), or by increasing the number of opportunities (the denominator).

For example, we might be interested in the number of typing errors in this post. The DPMO metric might be 1,000,000 x Errors/Total Words. But if this number didn’t look good enough, I might also use 1,000,000 x Errors/Total Letters, or 1,000,000 x Errors/Total Characters, counting spaces and punctuation.

The solution to metrics gaming is to use metrics to guide improvement, not to measure the performance of people. Metrics should be limited to those numbers that quantify an important outcome (Y metrics), or quantify an input that is critical to the quality of the outcome (a CTQ or X metric). The reason for quantifying these things is to discover, validate, and use a transfer function—such as Y = f(x), a model of the cause-and-effect relationship—to guide improvement planning and activity. When metrics serve a useful purpose such as this, the tendency to manipulate and game them is, if not eliminated, at least reduced.

Discuss

About The Author

Tom Pyzdek’s picture

Tom Pyzdek

Thomas Pyzdek’s career in business process improvement spans more than 50 years. He is the author more than 50 copyrighted works including The Six Sigma Handbook (McGraw-Hill, 2003). Through the Pyzdek Institute, he provides online certification and training in Six Sigma and Lean.

Comments

Exaggerating

I was NOT exaggerating. I documented FACTS. I learned use of 6 Sigma at Motorola where I worked 1989 to 2002. I presented the FACTS to managers in the quarterly quality presentations. As a safety professional I KNOW facts from fiction

Gaming

I always cite Brian Joiner who wrote in Fourth Generation Management that there are three things you can do to hit your numbers:

1) Improve the system

2) Distort the system

3) Distort the numbers

#2 and #3 are easiest in the short term, sadly. I've seen a lot of it in my career. 

Another side effect of the data talk is the unfortunate quote "If you can't measure it, you can't manage it." That's often wrongly attributed to Dr. Deming, when he taught the exact opposite - that some of the most important things in a business can't be measured.

Good Article

You hit the nail on the head, Tom! Are we going to use the data to whack people with, or are we going to use the numbers to help people improve and succeed? Gee, which one stands the better chance of getting us the outcomes we want? (Correct answer is: "Duh!")

I spent more than 30 years of

I spent more than 30 years of my life in high volume production/maintenance both a grunt and a supervisor. In both roles I counted on such metrics to evaluate first my performance and then others'. Metrics can only be "gamed" by those in charge being as they are the ones who set the parameters of the game. As a grunt (a term I use proudly), neither I nor my fellow trench-mates, had the liberty to change any part of the metric du jour. My only option was to either find the most efficient way of doing my particular task or not. While I chose the former path, some coworkers chose "not". We didn't have the option to change either the particular item to be produced nor how many were required to meet quota. We did have direct influence over quality. As a supervisor, I still didn't have the luxury of changing volume required and my influence over quality was converted to using the available employees in the most efficient manner possible. My point is that when the discussion concerning the use of metrics in production is limited to "other than Russian production from Hell" the whole notion of gaming metrics is relegated to upper management and rarely has any bearing on the floor-level employee. We were pretty much masters of our own employment destinies.