Featured Product
This Week in Quality Digest Live
Quality Insider Features
Gleb Tsipursky
Believing things will normalize soon, many companies are unprepared for new waves of restrictions
Harry Hertz
Communities of Excellence 2026 can help communities work together using a Baldrige-based, community-centric framework
Michael Weinold
GE’s exit from the lighting business is a warning to other players in the sector
Ken Voytek
Productivity will be even more critical as we recover from the current health and economic crisis
Jeffrey Phillips
Putting data to work

More Features

Quality Insider News
3D scans help Chicago Jet Group retrofit old Dassault Falcon avionics system
Real-time data collection and custom solutions for any size shop, machine type, or brand
Lloyd Instruments launches the LS5 high-speed universal testing machine
Measure diameter, ovality of wire samples, optical fibers and magnet wire, including transparent products
Training, tips and tricks, unboxing, and product videos provide additional information for users
How to develop an effective strategic plan and make the best major decisions in the context of uncertainty and ambiguity
Collect measurements, visual defect information, simple Go/No-Go situations from any online device
Laser scanning also used to help create safety covers for credit card readers
A complimentary webinar for novices to experts on May 27-28, 2020

More News

Steven Ouellette

Quality Insider

Blocking Out the Nuisance, Part One

You all know what happens when you make assumptions

Published: Tuesday, September 13, 2011 - 14:20

No, I am not writing about going, “Lalalalala! I can't hear you!” when someone is trying to tell you bad news. In the last couple of articles, we have been exploring how to properly perform research in industry. In this one, we will take a look at how you can handle variables in the real world that just get in the way, but still need to be dealt with. And there is a Really Important Thing partway through this article that will save you a lot of money.

So far we have done a good chunk of the research planning. When last we interfaced, we were planning an experiment to see how manipulating different process variables (the “treatments”) would affect the expansion of a complex polymer. But since it is the real world we are working in, we had also identified humidity as a variable that could affect expansion, a variable over which we had little control.


actstudyplando identifyandclassify

Figure 1: The research design process from Design of Experiments in Quality Engineering, by Jeffrey T. Luftig and Victoria S. Jordan (McGraw-Hill, 1998)
Click here for larger image.

Now the question was, “How are we going to handle humidity?” We have no plans to air-condition or dehumidify the factory floor, so it is not like we can reduce the humidity. It is a nuisance—it could affect things, but we really can’t control it.

As you might guess, this is not at all an unusual occurrence in the real world of business. So let’s talk about how to handle a situation like that.

Really important thing

So here is one of the things I really wanted to tell you about in the context of research design. Knowing this will save you money and be a big advantage over other researchers, as I promised in my first article on this subject.

There are three ways to control for nuisance variables. The worst one is to make assumptions. You just assume that the known factors won’t have an effect, or if they do it’s a small one, or if it’s not a small one, that it’s just an extrapolation of what you saw in the study.

This is a bad idea.

These known factors might have very large and unexpected effects on your dependent variable, rendering anything you think you are finding in your experiment misleading.

So for you technoids out there longing to do an ANOVA....

Let’s say That Guy Over There is trying to increase the critical performance measure “expansion” (higher is better). Historically the process has run at around 70-percent expansion. That Guy is very excited to design an experiment:

Figure 2: Expansion variables

He talks to his team, and asks them to identify three factors that they can control on the process that might affect expansion:

Figure 3: Three factors that might affect expansion

Obviously, he has decided to ignore this article (heck, he probably doesn’t even read my articles, the Philistine), so he goes forth and designs a full factorial with these factors and runs the experiment with n = 2 per combination. He places the data on the group hard drive, so you can download it, too.

He jumps right in and analyzes the experiment (totally not checking any test assumptions, of course) with an α = 0.05 and finds this:

Figure 4: Analysis of variance (ANOVA) ignoring humidity

Now first off, there is a subtle signal here that something is wrong—do you see it?

Take a look at the F-ratios. Recall that the F-statistic is:

It’s just the ratio of the variability we see between effects (the Mean Square for the different sources) and the variability we see within each combination (the Mean Square of the error). The expected value of F-ratios is 1 if there is no effect (all between variability is statistically the same as the within variability).

On the other hand, if there is an effect of the factor, the between variability should be larger than the within variability, leading to a ratio greater than 1. (Don’t understand why? Try reading “ANOVA by Any Other Name.”) Yet we see one F-ratio (the temp*speed interaction) that is fairly far below 1. This is a possible indication of some sort of confounding—a lot of variability going into the error term (the within-cell variance) making it big (146.565), as compared to the much smaller variance between the six means of the interaction (7.092). Of course this could be due to random variability, and since ANOVA is a directional test, we cannot make a statistical conclusion about it being too small, but it would be something to be suspicious about.

Well, That Guy Over There certainly doesn’t know to do this, so he proceeds although he’s clueless. He looks at that R2 (incorrectly) and sees 0.662 and assumes that he has made progress on the problem of expansion. He concludes that we can ignore everything except the main effect of temperature—everything else has a p-value above 0.05, his chosen α, so just run those wherever the operators want. Here is what happened with the expansion as temperature varied in the experiment.

Figure 5: The expansion as temperature varied in the experiment

Based on that, he selects the following setting as optimum:

Figure 6: Prediction of optimum for nonblocked experiment

 It looks like he is expecting to increase the expansion by about 7 percent. That rather large standard deviation means that there is a pretty big chunk of unexplained variability. In fact, that 7-percent increase in expansion is going to be hard to see over the random noise of the process.

Now, what he didn’t know, since he didn’t read this article, is that you have to identify every known potential factor and deal with it somehow. If he had simply asked, “What other factors could affect expansion?” his team would have immediately said, “Well, humidity might.” Or perhaps he knew about humidity, but since he can’t control it, he assumed that it would be a small effect.

You all know want happens when you make assumptions like that, right? It makes an ass out of your… umptions. Something like that.

Anyway, humidity did its own thing during this time (a kind of sinusoidal up and down pattern through the experiment), and he didn’t bother to record it on an engineering log, nor did he even randomize the order of the runs (which meant that humidity’s effects were confounded with the process settings, especially temperature). I guess he figures that order is as random as anything else.

So now you decide to take a crack at it, and, having read this article, you do ask what else can affect expansion, and you find out that the humidity is a likely culprit.

You could then use the second technique to handle nuisance variables: limit or control the factor. I suppose you could perform the experiment in the lab at some level of controlled humidity, or maybe on the factory floor during a time of some constant humidity level, but what would you learn that you could apply to the actual process? This technique leaves you open to questions about the external validity of anything you find.

Or, you could use the third technique, blocking. This underused technique is very powerful and will save you lots of money and time and get you better results.

In blocking, we have a factor that we think affects the dependent variable, but we don’t care to include it in the experiment. You know humidity is going to vary, and it is going to affect the expansion somewhat. You don’t care to prove that; you just need to control for it. (However, if you think the factor interacts with any of the other main effects, you would need to incorporate it into the experiment.) The amazing thing is that we don’t have to increase sample size to add a block. If that isn’t better than getting a free lunch, I don’t know what is.

So you design the same experiment, adding in humidity as a blocked factor. Logistically, you choose to run the first set of process settings (in random order) at a low-humidity time and the second set (in random order) at a high-humidity time.

Figure 7: Experiment variables adding humidity as a blocked factor

You run the experiment and also post the results on the group hard drive, which you can download here.

Go ahead and try analyzing that; see if you come up with something different than That Guy Over There. I’ll give you the analysis in Part 2.


About The Author

Steven Ouellette’s picture

Steven Ouellette

Steven Ouellette is the Lead Projects Consultant in the Office for Performance Improvement at the University of Colorada, Boulder. He has extensive experience implementing the systems that allow companies and organizations to achieve performance excellence, as well as teaching Master's-level students the tools used in BPE. He is the co-editor of Business Performance Excellence with Dr. Jeffrey Luftig. Ouellette earned his undergraduate degree in metallurgical and materials science engineering at the Colorado School of Mines and his Masters of Engineering from the Lockheed-Martin Engineering Management Program at the University of Colorado, Boulder.