The Paradox of Acceptance Sampling

We keep trying to find the needle in a haystack

Published: Tuesday, October 2, 2018 - 11:03

Aerospace standard AS9138—“Quality management systems statistical product acceptance requirements” was issued this year (2018), a few years after its accompanying guidance materials in section 3.7 of the International Aerospace Quality Group’s (IAQG) Supply Chain Management Handbook. The new aerospace standard supersedes the aerospace recommended practice of ARP9013 and, related to MIL-STD-105 (ANSI/ASQ Z1.4), claims to shift focus from the producer’s risk to the consumer’s risk with sampling plans having an acceptance number of zero (c=0).

Somewhere along this evolutionary path, the sampling procedures of MIL-STD-105 have fallen out of favor, even though the consumer risks of MIL-STD-105 at their designed lot tolerance percent defective (LTPD) point are superior to most plans found within AS9138.

Although c=0 sampling plans resolve the moral dilemma of accepting material when nonconforming units are observed in the sample, one should not assume they provide better consumer protection. The early pioneers Harold Dodge, George Edwards, Thornton Fry, Edward Molina, Paul Olmstead, Harry Romig, Walter Bartky, Mary Torrey, Walter Shewhart, and others, did not lead us astray with their c>0 acceptance sampling plans, and in fact, they debated the merits of c=0 plans then adopted the c>0 approach in order to establish fair play among large and small suppliers providing exactly the same item.

In October 1929, Harold F. Dodge and Harry G. Romig wrote their landmark paper, “A Method of Sampling Inspection” in the Bell System Technical Journal, and their sampling plans set consumer risk at 0.10. Their first requirement was to establish the idea of a LTPD point, and in 1924 the term “consumer’s risk” was established. Their second requirement was to minimize total inspection costs when considering screening costs. In 1941, Dodge and Romig issued single and double sampling inspection tables that are, in fact, indexed by consumer risk. Options for c=0 plans are clearly shown in their early work, but these plans hardly ever optimize total inspection costs.

The role of acceptance sampling in decision making

Before moving forward with this discussion, it is fundamental to remind ourselves of the role acceptance sampling plays in decision making. Broadly speaking, the purpose will be either to judge the quality of isolated lots of uninspected product (product control), or to judge the process that is producing the product (process control). This article will address both of these purposes as they relate to c=0 and c>0 acceptance sampling plans. Of particular importance is the paradox present with c=0 acceptance sampling procedures and the potential danger to the novice.

The graph of figure 1 shows the c=0 sample size required to detect nonconforming units in lots of varying quality. For simplicity, the sample size curve is based on the binomial distribution and assumes a 10-percent consumer’s risk. This risk level corresponds with the LTPD point on an operating characteristic curve, and is the probability of accepting the lot when the sample failed to find existing nonconforming units. The curve in figure 1 can therefore be thought of as the line that passes through all possible c=0 operating characteristic curves at the LTPD point.

The associated formula is shown within the graph for reference and can also be used to demonstrate that reducing consumer risk will also result in larger sample sizes. For example, reducing consumer risk from 10 percent to 5 percent will result in sample sizes that are 30-percent larger than those illustrated in figure 1. A graph of 5-percent consumer risk would correspond to the “rejectable quality level” on an operating characteristic curve as defined in AS9138.

Sampling inspection

The graph illustrates the counterintuitive nature of sampling inspection. That is, good-quality lots require larger sample sizes than bad-quality lots in order to detect the nonconforming units. For lots that are better than, say, 97-percent conforming, the sample size required rapidly approaches 100-percent inspection.

To illustrate the utility of this relationship, let’s suppose your supplier has been providing you parts that were injection-molded in a 20-cavity mold. During monthly quality system checks, the supplier discovered that one of the 20 cavities was not being heated properly. Your supplier believes he may have shipped you nonconforming material, but he is not sure. How many parts should your inspector look at in order to validate if this condition exists in your stockroom? Answer: Knowing the proportion conforming is 0.95, the sample size would be 45 parts, assuming a 10-percent consumer risk.

The mathematical relationship also clearly illustrates the limitations of acceptance sampling. For example, we can see that a 13-piece sample is effective for detecting nonconformance levels greater than 16 percent (0.86 proportion conforming). If we need to improve detectability, a 28-piece sample will detect 8-percent nonconforming product. A 56-piece sample will detect 4-percent nonconforming product, and a 114-piece sample is required to find 2-percent nonconforming product.

The relationship is not linear, and quickly the required sample size becomes equal to the lot size. Twice as much inspection does not provide twice as much protection. Inspection is a system of diminishing returns, with the largest cost benefit in the first 10 parts inspected. There simply is no way of finding a needle in the haystack by acceptance sampling and its inherent diminishing returns.

It becomes impractical to perform acceptance sampling for reliable detection of defect rates smaller than 3-percent nonconforming, and the truth is that all sampling plans will routinely pass a certain proportion of nonconforming material to stores without our awareness. On the other hand, acceptance sampling is an effective screen for detecting gross error levels.

To illustrate this point, the reader will note that the consumer risk (beta) shown in the numerator of the formula in figure 1 is the risk of accepting nonconforming material. Using the formula and inserting a beta risk of 0.50 (50%) in the numerator, and a proportion conforming of 0.95 in the denominator (5% nonconforming), the resulting sample size is the ever popular 13- or 14-piece sample. In other words, a 14-piece sample will allow a 5-percent nonconforming lot of material to pass to stores 50 percent of the time, even though zero nonconforming units were found in the sample. This plan corresponds to an equal risk point plan of AS9138.

The paradox within the quality profession is that we preach “parts per million” defect rates, and “zero defect” levels—yet we implement c=0 sampling plans and somehow believe material passing to stores is conforming because the sample was conforming. The emotional appeal of c=0 sampling plans can’t be denied, and the initial sample sizes required are relatively small by comparison, but the quality professional must remain mindful that AS9138 is not a substitute for the process control qualities of MIL-STD-105. While the intention of AS9138 is to make judgements regarding isolated lots (product control), the intention of MIL-STD-105 is to make judgements about a process that produces product (process control).

There are many improper comparisons in the literature, but here are the simple facts: Only a few plans within AS9138 can match the LTPD consumer protection provided by MIL-STD-105; AS9138’s focus is on isolated lot acceptance, and MIL-STD-105’s focus is on process control. AS9138 sacrifices the quality of process control using c=0 acceptance criterion in order to initially achieve smaller sample sizes; however, the total inspection labor required by AS9138 may actually be greater than MIL-STD-105 when considering screening costs.

Acceptance sampling of any type is not in harmony with “parts per million” nonconformance goals. The only way to attain low defect levels is through process improvement, and not inspection. The hard work needs to be done at the point of product generation, and once low nonconformance levels have been attained, the useless acceptance sampling for conformance to requirements must be eliminated.

This article focused primarily on the strengths and weaknesses of c=0 acceptance sampling procedures. In my next article, I will demonstrate the process control features of MIL-STD-105.

About The Author

Anthony Chirico

Anthony Chirico is a senior executive within the aerospace sector and has more than 30 years experience leading international quality assurance and supply chain organizations. Chirico holds a master’s degree in engineering from Rochester Institute of Technology in applied and mathematical statistics.