Blocking Out the Nuisance, Part Two

On the practical benefits of blocking

Published: Wednesday, September 14, 2011 - 10:37

Yesterday in “Blocking Out the Nuisance, Part One,” we saw the results of an experiment done by That Guy Over There that didn’t control for the environmental variable of humidity. There was a lot of variability in that experiment, so we didn’t see a whole lot of improvement to be made. I also gave you the data from a blocked design that you created which controlled for humidity. Let’s take a look at the analysis and see if we can find out anything more.

You perform the analysis at α = 0.05, test and pass the assumptions for analysis of variance (ANOVA), and find the following:

Figure 1: Analysis of variance (ANOVA) with humidity blocked

In the immortal words of the philosopher Neo, “Whoa.”

You see an entirely different result than That Guy Over There, don’t you? First off, with an adjusted R2 of 0.996 and an overall ω2 of 0.9979, you properly conclude that you have nailed it. You see a significant three-way interaction between all of the factors:

Figure 2: Temperature and speed interaction at feed = 1

Figure 3: Temperature and speed interaction at feed = 2

Figure 4: Temperature and speed interaction at feed = 3

This is exactly the same process as was tested in the first experiment. The reason we can detect more effects is that in this design, we are controlling for the different levels of humidity as a blocked factor. To get an idea of what is going on, here is an analysis of the same data but without including the blocked factor:

Figure 5: Analysis of variance (ANOVA) of blocked data without humidity
These are the exact same data as before, but the variability due to the humidity is unassigned, and so it goes into the error term.

To see what I mean, compare just these parts:

Figure 6: Blocked term and error

Do you see how the total variability is exactly the same, but how the error sum of the squares in the second one is the sum of the variability of the humidity and the error?

7949.362 + 25.765 = 7975.127

All we did by assigning the variability to the humidity is subtract it out of the error. (Note that we do also lose a degree of freedom from that error, but that is far outweighed by the huge decrease.) Since the error term is what we divide each effect’s mean square by in order to get our F-statistic, we are dividing by a lot smaller number, meaning that we get a lot bigger F-statistics.

For example, the F-statistic of the speed*feed interaction is 1.218 without the block (which is probably going to be tough to differentiate from 1.000 without a huge sample size), as opposed to 131.697 for the blocked analysis, which if there is any fairness in the cosmos will always be a significant effect. So, all else being equal (except for that drop in the degrees of freedom), it is a whole lot easier to see that the different effects are significant: There is a lot less noise over which to hear the process screaming out to you.

Practical benefits of blocking

What are the practical benefits of the blocked analysis?
• You have learned a lot more about the process, so you can choose better settings to maximize the expansion in the face of changing humidity.
• On the other hand, with the unblocked design, That Guy Over There missed significant factors, due to the humidity, and chose the wrong setting, due to not randomizing.

Let’s take a look at our optimum settings. Because we blocked humidity, we are assuming that there is no interaction, so the optimal settings are going to be the same for different levels of humidity, but they will result in different expansions depending on where the humidity is that day.

 Figure 7: Prediction of optimum for blocked experiment

Now compare this table in figure 7 to the table in figure 5. You can see that during times of low humidity, we predict a much higher expansion, and for times of high humidity, a lower expansion. In both cases, we have a lot less variability.

Let’s make a picture to compare our predictions:

Figure 8: Comparison of predictions for humidity blocked and unblocked experiments

The experiment without blocking has a lot higher variability since all that variability due to humidity is unexplained and going into the error term. This unexplained variability washed out effects that really were significant, misleading That Guy Over There into thinking that none of the settings except temperature had an effect. Therefore, the additional variation that is really caused by these process settings gets thrown into the expected variation, too. Even worse, since That Guy Over There didn’t randomize, and since the low temp setting just happened to mostly correspond with a low humidity time frame, he concludes that the best setting for temperatures was the low one, when in fact it was the high one.

This means that when That Guy Over There implements his settings, he will not get 79.013 on average across all the other settings and humidity like he expects, but something closer to 77.65. That expansion is about 10 points lower than the average you would expect with our optimum settings across the humidity range (about 88.7%).

By the way, if you are wondering why our lower prediction of 73.773 percent is below what he thinks he will get, remember that his prediction is averaged across all humidities and settings. Ours is just for the high humidity. In reality, using his settings when the humidity was high, we would expect to see 63.34 percent.

You also now have a powerful argument to go back to management and see if they will be willing to start air conditioning the shop floor. Your customers are not going to see those nice little distributions but will be getting expansions all across the range between the two of them as the humidity varies, so it is doubtful they will notice much of an improvement. Your process is capable of high expansion with very low variability if you could just control humidity. (Or perhaps find a way of making your process robust to the humidity variation, which is exactly what happened in the real experiment.)

Summing up variable identification

Blocking properly accounts for variables that might affect the output but that we don’t care to include in the experimental design, such as nuisance variables. Blocking allows us to more easily see significant effects without increasing the sample size. By identifying all potential factors, including ones we don’t plan on incorporating into the experiment, we can avoid making major errors that will doom us to failure even before we set up the first experimental run.

So don’t be That Guy Over There, running all sorts of experiments but not getting replicable results, mocked by those who read this article. Use blocking and succeed.

More on the next few steps coming up in my next column.

About The Author

Steven Ouellette

Steve Ouellette, ME, CMC started his career as a metallurgical engineer. It was during this time that his eyes were opened to find that business was not a random series of happenings, but that it could be a knowable enterprise. This continues to fascinate him to this day.

He started consulting in 1996 and is a Certified Management Consultant through the Institute for Management Consulting. He has worked in heavy and light industry, service, aerospace, higher education, government, and non-profits. He is the President of The ROI Alliance, LLC. His website can be found at steveouellette.com.

Steve has a black belt in aikido, a non-violent martial art, and spent a year in Europe on a Thomas J. Watson Fellowship studying the “Evolution, Fabrication, and Social Impact of the European Sword."