Featured Product
This Week in Quality Digest Live
Quality Insider Features
Libby Sander
Seven tips to boost well-being and productivity
Vanessa Bates Ramirez
That’s right—all 6.5 million articles
Jennifer Lauren Lee
After 30 years NIST is once again calibrating gear artifacts, which could improve electric cars
Matt Fieldman
It’s vital that we address social determinants of work
Lisa Wong Macabasco
Challenging stereotypes without sacrificing likability

More Features

Quality Insider News
US Dept. of Commerce issues seven grand challenges
1-nanometer axis resolution and less than +/– 0.002 mm line-form accuracy
New calibration software increases efficiency
An MIT researcher and her colleagues are looking to DNA to help guide the process
Requesting quotes for ‘Baldrige Reimagined’
AS-Schneider will present the Digital Valve Kit, customized injection quill solutions at ADIPEC 2022
Providing operators a simple upgrade or ability to switch between two controls on a single machine
From excess inventory and nonvalue work to $2 million in cost savings

More News

Steven Ouellette

Quality Insider

Blocking Out the Nuisance, Part One

You all know what happens when you make assumptions

Published: Tuesday, September 13, 2011 - 14:20

No, I am not writing about going, “Lalalalala! I can't hear you!” when someone is trying to tell you bad news. In the last couple of articles, we have been exploring how to properly perform research in industry. In this one, we will take a look at how you can handle variables in the real world that just get in the way, but still need to be dealt with. And there is a Really Important Thing partway through this article that will save you a lot of money.

So far we have done a good chunk of the research planning. When last we interfaced, we were planning an experiment to see how manipulating different process variables (the “treatments”) would affect the expansion of a complex polymer. But since it is the real world we are working in, we had also identified humidity as a variable that could affect expansion, a variable over which we had little control.

 

actstudyplando identifyandclassify


Figure 1: The research design process from Design of Experiments in Quality Engineering, by Jeffrey T. Luftig and Victoria S. Jordan (McGraw-Hill, 1998)
Click here for larger image.

Now the question was, “How are we going to handle humidity?” We have no plans to air-condition or dehumidify the factory floor, so it is not like we can reduce the humidity. It is a nuisance—it could affect things, but we really can’t control it.

As you might guess, this is not at all an unusual occurrence in the real world of business. So let’s talk about how to handle a situation like that.

Really important thing

So here is one of the things I really wanted to tell you about in the context of research design. Knowing this will save you money and be a big advantage over other researchers, as I promised in my first article on this subject.

There are three ways to control for nuisance variables. The worst one is to make assumptions. You just assume that the known factors won’t have an effect, or if they do it’s a small one, or if it’s not a small one, that it’s just an extrapolation of what you saw in the study.

This is a bad idea.

These known factors might have very large and unexpected effects on your dependent variable, rendering anything you think you are finding in your experiment misleading.

So for you technoids out there longing to do an ANOVA....

Let’s say That Guy Over There is trying to increase the critical performance measure “expansion” (higher is better). Historically the process has run at around 70-percent expansion. That Guy is very excited to design an experiment:


Figure 2: Expansion variables

He talks to his team, and asks them to identify three factors that they can control on the process that might affect expansion:


Figure 3: Three factors that might affect expansion

Obviously, he has decided to ignore this article (heck, he probably doesn’t even read my articles, the Philistine), so he goes forth and designs a full factorial with these factors and runs the experiment with n = 2 per combination. He places the data on the group hard drive, so you can download it, too.

He jumps right in and analyzes the experiment (totally not checking any test assumptions, of course) with an α = 0.05 and finds this:


Figure 4: Analysis of variance (ANOVA) ignoring humidity


Now first off, there is a subtle signal here that something is wrong—do you see it?

Take a look at the F-ratios. Recall that the F-statistic is:

It’s just the ratio of the variability we see between effects (the Mean Square for the different sources) and the variability we see within each combination (the Mean Square of the error). The expected value of F-ratios is 1 if there is no effect (all between variability is statistically the same as the within variability).

On the other hand, if there is an effect of the factor, the between variability should be larger than the within variability, leading to a ratio greater than 1. (Don’t understand why? Try reading “ANOVA by Any Other Name.”) Yet we see one F-ratio (the temp*speed interaction) that is fairly far below 1. This is a possible indication of some sort of confounding—a lot of variability going into the error term (the within-cell variance) making it big (146.565), as compared to the much smaller variance between the six means of the interaction (7.092). Of course this could be due to random variability, and since ANOVA is a directional test, we cannot make a statistical conclusion about it being too small, but it would be something to be suspicious about.

Well, That Guy Over There certainly doesn’t know to do this, so he proceeds although he’s clueless. He looks at that R2 (incorrectly) and sees 0.662 and assumes that he has made progress on the problem of expansion. He concludes that we can ignore everything except the main effect of temperature—everything else has a p-value above 0.05, his chosen α, so just run those wherever the operators want. Here is what happened with the expansion as temperature varied in the experiment.


Figure 5: The expansion as temperature varied in the experiment


Based on that, he selects the following setting as optimum:


Figure 6: Prediction of optimum for nonblocked experiment

 It looks like he is expecting to increase the expansion by about 7 percent. That rather large standard deviation means that there is a pretty big chunk of unexplained variability. In fact, that 7-percent increase in expansion is going to be hard to see over the random noise of the process.

Now, what he didn’t know, since he didn’t read this article, is that you have to identify every known potential factor and deal with it somehow. If he had simply asked, “What other factors could affect expansion?” his team would have immediately said, “Well, humidity might.” Or perhaps he knew about humidity, but since he can’t control it, he assumed that it would be a small effect.

You all know want happens when you make assumptions like that, right? It makes an ass out of your… umptions. Something like that.

Anyway, humidity did its own thing during this time (a kind of sinusoidal up and down pattern through the experiment), and he didn’t bother to record it on an engineering log, nor did he even randomize the order of the runs (which meant that humidity’s effects were confounded with the process settings, especially temperature). I guess he figures that order is as random as anything else.

So now you decide to take a crack at it, and, having read this article, you do ask what else can affect expansion, and you find out that the humidity is a likely culprit.

You could then use the second technique to handle nuisance variables: limit or control the factor. I suppose you could perform the experiment in the lab at some level of controlled humidity, or maybe on the factory floor during a time of some constant humidity level, but what would you learn that you could apply to the actual process? This technique leaves you open to questions about the external validity of anything you find.

Or, you could use the third technique, blocking. This underused technique is very powerful and will save you lots of money and time and get you better results.

In blocking, we have a factor that we think affects the dependent variable, but we don’t care to include it in the experiment. You know humidity is going to vary, and it is going to affect the expansion somewhat. You don’t care to prove that; you just need to control for it. (However, if you think the factor interacts with any of the other main effects, you would need to incorporate it into the experiment.) The amazing thing is that we don’t have to increase sample size to add a block. If that isn’t better than getting a free lunch, I don’t know what is.

So you design the same experiment, adding in humidity as a blocked factor. Logistically, you choose to run the first set of process settings (in random order) at a low-humidity time and the second set (in random order) at a high-humidity time.


Figure 7: Experiment variables adding humidity as a blocked factor


You run the experiment and also post the results on the group hard drive, which you can download here.

Go ahead and try analyzing that; see if you come up with something different than That Guy Over There. I’ll give you the analysis in Part 2.

Discuss

About The Author

Steven Ouellette’s picture

Steven Ouellette

Steve Ouellette, ME, CMC started his career as a metallurgical engineer. It was during this time that his eyes were opened to find that business was not a random series of happenings, but that it could be a knowable enterprise. This continues to fascinate him to this day.

He started consulting in 1996 and is a Certified Management Consultant through the Institute for Management Consulting. He has worked in heavy and light industry, service, aerospace, higher education, government, and non-profits. He is the President of The ROI Alliance, LLC. His website can be found at steveouellette.com.

Steve has a black belt in aikido, a non-violent martial art, and spent a year in Europe on a Thomas J. Watson Fellowship studying the “Evolution, Fabrication, and Social Impact of the European Sword."