Featured Product
This Week in Quality Digest Live
Statistics Features
Donald J. Wheeler
The more you know, the easier it becomes to use your data
Scott A. Hindle
Part 7 of our series on statistical process control in the digital era
Donald J. Wheeler
How you can filter out noise
Scott A. Hindle
Part 6 of our series on SPC in a digital era
Douglas C. Fair
Part 5 of our series on statistical process control in the digital era

More Features

Statistics News
How to use Minitab statistical functions to improve business processes
New capability delivers deeper productivity insights to help manufacturers meet labor challenges
Day and a half workshop to learn, retain, and transfer GD&T knowledge across an organization
Elsmar Cove is a leading forum for quality and standards compliance
InfinityQS’ quality solutions have helped cold food and beverage manufacturers around the world optimize quality and safety
User friendly graphical user interface makes the R-based statistical engine easily accessible to anyone
Collect measurements, visual defect information, simple Go/No-Go situations from any online device
Good quality is adding an average of 11 percent to organizations’ revenue growth
Ability to subscribe with single-user minimum, floating license, and no long-term commitment

More News

David Currie


Metrics: The Good, the Bad, and the Ugly

Part one: the good

Published: Monday, October 22, 2018 - 11:03

Metrics are an important part of an effective quality management system (QMS). They are necessary to understand, validate, and course-correct the QMS. They should be used to verify that it is achieving the goals and objectives defined by management. In an ISO 9001 system, metrics must be available to assess risk, and validate changes made to the QMS and individual processes. Metrics are also used to validate improvement and verification of corrective action implementation during the management review.

I have seen and used many metrics in the past, and in my experience not all metrics are equally good; in fact, many are totally inappropriate for the purpose for which they are being used. This article, the first in a three-part series, will help readers distinguish good metrics from bad—or as the title suggests, the downright ugly. Once the characteristics of a good metric are known, a bad metric can be converted into a good metric. This series is divided into three parts: Part one explains what a good metric is, part two identifies bad metrics and explains how to convert them, and part three looks at ugly metrics and explains why they have no hope for conversion.

The good

A good metric is derived from clearly defined process measurements, carefully collected, and properly combined to provide a clear and understandable representation of a process. A good metric supports the goals and objectives of the organization and helps focus the QMS toward achieving these goals. This definition presumes that the underlying process of data collection is also well understood, so that a distinction can be made between problems in the measured process and problems in the data collection process. The data used in the metric must contain sufficient detail to identify problem areas, prioritize those with the greatest impact, and provide a starting point for an investigation into those problem areas.

In short, a metric used to measure process performance should have the following characteristics:
• The metric supports the goals and objectives of the quality system.
• The data contain sufficient detail to allow analysis of specific defects.
• Data are carefully collected, and checked for accuracy and completeness.
• Data are combined in a way that clearly represents the process.
• The data collection process is clearly understood.
• There is a clear relationship between the process and the data being used.
• The review interval of the metric matches the response time for corrections.
• The metric results in process improvement and overall cost savings.

A sample good metric

The following is an example of a good metric. It is important to distinguish a good metric from a bad one by a careful examination of details. The metric collected is the result of 100-percent final inspection and functional check of a final assembly, in this case a tool box.

The metric we tracked was the number of units rejected divided by the number of units completed per day and summed for the month. The criteria used to inspect the units was a combination of workmanship, form and fit, and a functional check of the unit. The functional check included locking and unlocking and drawer function. Form and fit looked at the clearances around the drawers to ensure they were even and not interfering. The workmanship included everything related to paint, welding, handling (dents and scratches), etc.

The engineering department did not define criteria, so it was up to the quality group to set standards of workmanship. This was done using customer complaints to define what customers were willing to accept and not accept. The reason that this metric was considered a good metric, is that it was responsive to actions taken, and could be used to clearly see the effect of different issues that caused peaks and valleys on the chart, both positive and negative.

An example of a good metric

The data for this example were collected for the express purpose of the metric, then reviewed for accuracy and understanding by the metric's user. Defects were defined by workmanship standards to ensure that criteria were consistently applied, not only in terms of the defect definition, but also the process from which the defect originated. New defects were carefully identified for inclusion in the workmanship standards and investigated for their originating process. Training was conducted to ensure that inspectors clearly understood the workmanship standards. In addition, an independent verification was performed to confirm that all units produced were inspected. This met the criteria of a good metric in the category of “carefully collected, and checked for accuracy and completeness.”

The data from the 100-percent inspection was collected and charted as a percent of units completed. The numerator of the statistic (rejects) was a clear subset of the denominator (total inspected). This is a critical concept in assessing the impact of corrective actions and identifying problems using the metric. Knowledge of the problems encountered, and actions taken relative to the overall process, allow the annotation of key points on the charted data. Although very simple, it is clear that the data collected were being used to improve and monitor the process, as all useful metrics should be.

The goal of this organization is customer delight. This objective is met by translating customer criteria and complaints into the inspection criteria at final inspection. Customer delight is measured directly by the customer using feedback sheets. The top half of the sheets are completed during final assembly by each operator who works on the unit. The sheets are packed in the bottom drawer and sent with each product produced. The bottom half of the form is a simple series of seven questions regarding the product. The customer completes the bottom half, giving yes or no answers along with any comments, and returns it postage paid.

The feedback from this process is reviewed and scored simply: Negative comments are scored (–1), positive (+ 1), and no response (0). All comments are collected and are added to the customer complaint log, if negative. This becomes a second metric, measuring overall customer satisfaction. The two metrics work well together to focus the attention of the QMS on improvements to both product and process to achieve the company goal. All of the various data collection processes—inspection, customer complaints, and customer satisfaction survey—use the same set of defect definitions and process source information so that a clear link exists between the different metrics.

The response of the metric to the review interval is appropriate in that the effect of corrections made in one review interval can be clearly reflected in the next review. This can be seen by being able to recognize the rise of specific defects and the effort to correct them, as well as significant changes to the process.

In part two, we will look at a bad metric and consider how to change it to a useful, good metric.


About The Author

David Currie’s picture

David Currie

David Currie is a quality professional with a broad background of experience in the nuclear (ANSI N45.2), commercial (ISO 9001), automotive (QS-9000), aerospace (AS 9100), and defense (MIL-Q-9858) quality systems.