Featured Product
This Week in Quality Digest Live
Statistics Features
Minitab Inc.
Passing the three FDA stage goals
David Currie
Part two: the bad
Donald J. Wheeler
How’s that classification scheme working out for you?
Anthony Chirico
We owe a debt of gratitude to Tippett and other pioneers who put ‘engineering’ into quality engineering.
Minitab Inc.
There’s a Goldilocks balance with the number of predictors to include

More Features

Statistics News
Provides accurate visual representations of the plan-do-study-act cycle
SQCpack and GAGEpack offer a comprehensive approach to improving product quality and consistency
Ask questions, exchange ideas and best practices, share product tips, discuss challenges in quality improvement initiatives
Strategic investment positions EtQ to accelerate innovation efforts and growth strategy
Satisfaction with federal government reaches a four-year high after three years of decline
TVs and video players lead the pack, with internet services at the bottom
Using big data to identify where improvements will have the greatest impact
Includes all the tools to comply with quality standards and reduce variability

More News

David Currie

Statistics

Metrics: The Good, the Bad, and the Ugly

Part one: the good

Published: Monday, October 22, 2018 - 11:03

Metrics are an important part of an effective quality management system (QMS). They are necessary to understand, validate, and course-correct the QMS. They should be used to verify that it is achieving the goals and objectives defined by management. In an ISO 9001 system, metrics must be available to assess risk, and validate changes made to the QMS and individual processes. Metrics are also used to validate improvement and verification of corrective action implementation during the management review.

I have seen and used many metrics in the past, and in my experience not all metrics are equally good; in fact, many are totally inappropriate for the purpose for which they are being used. This article, the first in a three-part series, will help readers distinguish good metrics from bad—or as the title suggests, the downright ugly. Once the characteristics of a good metric are known, a bad metric can be converted into a good metric. This series is divided into three parts: Part one explains what a good metric is, part two identifies bad metrics and explains how to convert them, and part three looks at ugly metrics and explains why they have no hope for conversion.

The good

A good metric is derived from clearly defined process measurements, carefully collected, and properly combined to provide a clear and understandable representation of a process. A good metric supports the goals and objectives of the organization and helps focus the QMS toward achieving these goals. This definition presumes that the underlying process of data collection is also well understood, so that a distinction can be made between problems in the measured process and problems in the data collection process. The data used in the metric must contain sufficient detail to identify problem areas, prioritize those with the greatest impact, and provide a starting point for an investigation into those problem areas.

In short, a metric used to measure process performance should have the following characteristics:
• The metric supports the goals and objectives of the quality system.
• The data contain sufficient detail to allow analysis of specific defects.
• Data are carefully collected, and checked for accuracy and completeness.
• Data are combined in a way that clearly represents the process.
• The data collection process is clearly understood.
• There is a clear relationship between the process and the data being used.
• The review interval of the metric matches the response time for corrections.
• The metric results in process improvement and overall cost savings.

A sample good metric

The following is an example of a good metric. It is important to distinguish a good metric from a bad one by a careful examination of details. The metric collected is the result of 100-percent final inspection and functional check of a final assembly, in this case a tool box.

The metric we tracked was the number of units rejected divided by the number of units completed per day and summed for the month. The criteria used to inspect the units was a combination of workmanship, form and fit, and a functional check of the unit. The functional check included locking and unlocking and drawer function. Form and fit looked at the clearances around the drawers to ensure they were even and not interfering. The workmanship included everything related to paint, welding, handling (dents and scratches), etc.

The engineering department did not define criteria, so it was up to the quality group to set standards of workmanship. This was done using customer complaints to define what customers were willing to accept and not accept. The reason that this metric was considered a good metric, is that it was responsive to actions taken, and could be used to clearly see the effect of different issues that caused peaks and valleys on the chart, both positive and negative.

An example of a good metric

The data for this example were collected for the express purpose of the metric, then reviewed for accuracy and understanding by the metric's user. Defects were defined by workmanship standards to ensure that criteria were consistently applied, not only in terms of the defect definition, but also the process from which the defect originated. New defects were carefully identified for inclusion in the workmanship standards and investigated for their originating process. Training was conducted to ensure that inspectors clearly understood the workmanship standards. In addition, an independent verification was performed to confirm that all units produced were inspected. This met the criteria of a good metric in the category of “carefully collected, and checked for accuracy and completeness.”

The data from the 100-percent inspection was collected and charted as a percent of units completed. The numerator of the statistic (rejects) was a clear subset of the denominator (total inspected). This is a critical concept in assessing the impact of corrective actions and identifying problems using the metric. Knowledge of the problems encountered, and actions taken relative to the overall process, allow the annotation of key points on the charted data. Although very simple, it is clear that the data collected were being used to improve and monitor the process, as all useful metrics should be.

The goal of this organization is customer delight. This objective is met by translating customer criteria and complaints into the inspection criteria at final inspection. Customer delight is measured directly by the customer using feedback sheets. The top half of the sheets are completed during final assembly by each operator who works on the unit. The sheets are packed in the bottom drawer and sent with each product produced. The bottom half of the form is a simple series of seven questions regarding the product. The customer completes the bottom half, giving yes or no answers along with any comments, and returns it postage paid.

The feedback from this process is reviewed and scored simply: Negative comments are scored (–1), positive (+ 1), and no response (0). All comments are collected and are added to the customer complaint log, if negative. This becomes a second metric, measuring overall customer satisfaction. The two metrics work well together to focus the attention of the QMS on improvements to both product and process to achieve the company goal. All of the various data collection processes—inspection, customer complaints, and customer satisfaction survey—use the same set of defect definitions and process source information so that a clear link exists between the different metrics.

The response of the metric to the review interval is appropriate in that the effect of corrections made in one review interval can be clearly reflected in the next review. This can be seen by being able to recognize the rise of specific defects and the effort to correct them, as well as significant changes to the process.

In part two, we will look at a bad metric and consider how to change it to a useful, good metric.

Discuss

About The Author

David Currie’s picture

David Currie

David Currie is a quality professional with a broad background of experience in the nuclear (ANSI N45.2), commercial (ISO 9001), automotive (QS-9000), aerospace (AS 9100), and defense (MIL-Q-9858) quality systems.