The engineer came into the statistician’s office and asked, “How can I compare a couple of averages? I have 50 values from each machine and want to compare the machines.”

The statistician answered, “That’s easy. We can use a two-sample t-test.”

“How would that work?” asked the engineer.

“We compute a t-statistic based on the difference between the averages, and if the p-value for the t-statistic is small, we’ll have a significant difference,” replied the statistician.

“So a small p-value means we have a big difference?”

“Not exactly. The size of the difference is always relative. It depends on the background variation. A significant difference is one that is detectable in spite of the background noise present. The actual difference may be either large or small. And a small difference may be of no practical importance,” said the statistician.

“So why not call it a detectable difference rather than a significant difference?”

“That’s just the language of statistics. We can’t afford to make things too clear because if we did, people might not seek our help.”

The engineer thought for a bit. “OK, I think I wasn’t clear about what I needed earlier. I have three machines I want to compare. How would we do that?”

“We can do that using an analysis of variance.”

“But I want to compare the averages!”

“Yes, we compare the averages by analyzing the variance.”

“How can that work?”

“We can compare the averages by analyzing the variance because the averages and variances are independent of each other.”

“That doesn’t make sense.”

“Oh, yes it does,” said the statistician. “Let me show you.”

The statistician then took the engineer’s data, entered them into the computer, and obtained the one-way ANOVA table shown in Figure 1.

Figure 1: One-way ANOVA

The statistician gave this table to the engineer and said, “With a p-value of 5 parts per billion, we have a highly significant result.”

The engineer responded, “I don’t know what all those numbers mean, but I agree that the p-value looks pretty small. So does that mean we have a big difference?”

“Not exactly. This small p-value tells us that the three averages are not all the same. One or more of these three averages is detectably different from the overall grand average.”

“So how do I find out which averages are different from each other?”

“For that we’ll need to make some pairwise comparison tests.”

Here the engineer stood up and said, “Thanks for the help. I’ll have to get back to you later.”

He then went down the hall to a colleague’s office. As the engineer described the runaround in the statistician’s office, his colleague suggested plotting the data on a process behavior chart. With three subgroups of size 50, his colleague recommended an “average and standard deviation chart.”

Figure 2: Average and standard deviation chart

With two averages outside the limits, we have detectable differences between all three averages. Machine One averages about 1.7 ten-thousandths of an inch more than Machine Two, and Machine Two averages about 1.6 ten-thousandths of an inch more than Machine Three.

Moreover, Machine Three has roughly twice as much variation as the other two machines (which is something you will not discover with ANOVA.)

The engineer said, “Now that is what I wanted to know. The differences in these averages aren’t large enough to be of any practical importance. But the excessive variation in Machine Three is a problem that needs to be addressed.”

To learn more about this problem, his colleague suggested placing the data from these machines on three XmR charts. When they did this, they got Figure 3.

Figure 3: XmR charts for three machines

While Machine One and Machine Two had a few points outside their limits, Machine Three is qualitatively different from the other two. The large oscillations, plus the two long runs below and above the central line, tell us that Machine Three is in serious trouble. Although assignable causes are affecting all three machines, they are affecting Machine Three much more severely.

Now the engineer had both the answer to his initial question plus the additional insight needed to fix his operation. Moreover, he had the graphs to communicate his findings to others so they could also understand the problem.

By showing the averages, standard deviations, and original data, process behavior charts like those in figures 2 and 3 make it much easier for users to make sense of their data. As a result, they are much more likely to come to the right conclusions—unlike the engineers in the next example.

Analysis without understanding

A European lab carried out an experiment to compare the effectiveness of two antiperspirant compounds. Each test subject would have the two compounds applied to their forearms. Of course, they randomized which compound was applied to each subject’s right and left arm. After a specified time in a controlled environment, the amount of perspiration on each arm was measured by means of galvanic skin response.

They put the 44 data in the computer, got their ANOVA table, and found a significant result. They then wrote up a report saying that Compound A was a better antiperspirant than Compound B.

When this report reached corporate headquarters, their boss didn’t understand the ANOVA table. So she put the 44 data on an XmR chart using the order in which they were listed on the spreadsheet: Subject 1, right arm; Subject 1, left arm; Subject 2, right arm; Subject 2, left arm, etc. When she finished the chart, she had a two-point pattern that repeated itself 22 times. For every one of the 22 subjects, the right arm perspired more than the left arm. (Everyone in the test panel was right-handed.) Moreover, this difference between right arms and left arms was the only signal within these data.

So what was the significant result from the ANOVA table? The random assignment of compounds to right and left arms wasn’t balanced. Compound A was used on more left arms than Compound B. This made Compound A look better.

After adjusting for the difference between the arms, there was no detectable difference between the two compounds. The boss was right. The data didn’t support the conclusions in the report. Recognizing the qualitative signals is just as important as finding the quantitative ones.

The purpose and objective of analysis is to discover and communicate that which is contained within your data. Graphs have always constituted the most powerful and effective way of doing this. Numerical analysis techniques may complement histograms, running records, and process behavior charts, but they can never replace them. The heavy lifting will always be done by the graphs. Creating a clear and easily understood graph that reveals the message within the data has always been the key to effective data analysis. Graphs help you put your data in context—and the first principle of data analysis is that no data have meaning apart from their context.

Donald Wheeler’s complete “Understanding SPC” seminar may be streamed for free. For details, see spcpress.com; for an example, see this column in Quality Digest.