Featured Product
This Week in Quality Digest Live
Quality Insider Features
Jason V. Barger
How to help your people and engage with them remotely, in hybrid and in-person settings
Innovating Service With Chip Bell
You will need all the half-baked, slightly wild ideas you can get
Dirk Dusharme @ Quality Digest
At the top of the list: Trust
John Hayes
Automation can lead to better paying, more fulfilling jobs
Sara Adams
Here’s how you can avoid making them

More Features

Quality Insider News
Update offers primary support of QIF 3.0 with bi-directional translation and validation of the MBD portion
The reshoring move will simplify operations
Alerts scheduled by actual use of the machine rather than number of days. See at PackExpo, Sept. 27–29, 2021.
Additional option for interfacing and controlling innovative mobile surface measurement system cuts implementation costs in half
Is the future of quality management actually business management?
Booth No. 1120, Donald E. Stephens Convention Center, Rosemont, Illinois, Oct. 26–28, 2021

More News

Matt Treglia

Quality Insider

Understanding Design of Experiments

Common questions and misconceptions

Published: Thursday, March 12, 2015 - 17:22

Design of experiments (DOE) is an approach used in numerous industries for conducting experiments to develop new products and processes faster, and to improve existing products and processes. When applied correctly, it can decrease time to market, decrease development and production costs, and improve quality and reliability.

DOE is much more rigorous than traditional methods of experimentation such as one-factor-at-a-time and expert trial-and-error. This rigor allows practitioners to explicitly model the relationships among the numerous variables in any system, make more informed decisions at each stage of the problem-solving process, and ultimately arrive at better solutions in less time.

DOE is a powerful method that can seem deceptively easy, but in reality it takes significant know-how to make it work reliably. Most unsuccessful attempts to apply DOE can be attributed to one of a handful of pitfalls. In addition to knowledge of statistical methods, the keys to making it work are discipline and effective communication between the statistician and the scientists, engineers, and managers on the project team who best understand the product or process.

For quality and process improvement professionals, the merits of DOE are old hat, and any discussion about DOE will likely revolve around technical details such as the pros and cons of Taguchi methods or the value of estimating quadratic effects. We take for granted that DOE is indispensable, and we forget that most managers and engineers are unaware of the value of DOE or—even worse—unaware that it exists at all. In fact, there are a surprising number of organizations, even R&D organizations, that run experiments all day, every day, and they use one-factor-at-a-time methods!

When talking about DOE for the first time with various unenlightened co-workers, clients, and, of course, attendees over the years, I’ve found that the crux of the conversation consists of addressing some common questions and misconceptions. Here is a compiled a list of these questions and misconceptions, along with some basic answers.  I hope it will help you the next time you find yourself talking with someone who is skeptical about DOE.

What is DOE, and why should my organization use it?

DOE is a structured approach for conducting experiments. It’s useful in product development, process development, and process improvement. Depending on the problem, benefits can include faster time to market, lower development costs, lower operating costs, and lower cost of poor quality.

DOE can be applied in any situation where you need to manipulate several independent variables (factors) in order to optimize one or more dependent variables. Consider as an example the bread-baking process. Independent variables would include the proportions of the various ingredients, baking time, and baking temperature. Dependent variables would include the texture, flavor, density, and cost of the finished product, as well as the throughput of the baking process. Hence, DOE could be used to determine the percentage of water added, baking time, and baking temperature that create the optimum flavor while maintaining a specified minimum throughput.

How does DOE work?

To illustrate how a typical DOE works, consider the simple experimental design shown in the first table below. It consists of eight runs in which four factors (A, B, C, and D) are each varied over two levels. Rather than “high” and “low,” the levels are typically coded as +1 and –1, as shown in the second table.

Then, after the experiment has been run, the data collected are analyzed to quantify the effect that each of the factors A, B, C, and D has on the dependent variable(s). Based on the conclusions of that experiment, additional rounds of experimentation are then performed to arrive at the optimum levels for each of those factors.

How is it possible to change more than one factor at a time?

In the example above, in going from one run to the next, notice that more than one factor are changed. In high school science class, we all learned that in order to evaluate a factor we need to hold every other variable constant and change only the factor under study. This implies that changing more than one factor at a time will make it impossible to distinguish the effects of one factor from the effects of another. That, however, is simply not true. As long as the factors are varied independently of one another (i.e., no two columns are correlated with each other), the effect of each factor can be quantified independently.

What’s wrong with using the one-factor-at-a-time approach?

As the name would imply, the one-factor-at-a-time (OFAT) approach is perfectly valid when testing only one factor. But what happens when there are two or more factors?

If we are evaluating two factors (say, A and B), then certain drawbacks in the OFAT method become apparent. If we apply the same logic we used in science class, then we would start with a control condition, let’s say with A and B both set to –1 (the low level). Then, to evaluate factor A, we would hold factor B constant at –1 and set factor A to +1. Then, in order to evaluate factor B, we would hold factor A constant at –1 and set factor B to +1. This design is shown in the table and the graph below.

An obvious disadvantage of this design is that the area in the upper right hand corner of the graph, corresponding to A = +1 and B = +1, is completely excluded from the study. This means that not only are we not considering that particular factor level combination (which may yield great results) but we’re also not able to measure any interactive effect between the two factors. In other words, we only know the effect of A when B is held at its low level, and we only know the effect of B when A is held at its low level.

In running an OFAT experiment, we’re implicitly assuming that the effects of A and B are independent of one another. This will often result in bad decision making because, in many real-world systems, factors don’t act independently. For example, the effect of quantity of sunlight on plant growth depends on the quantity of water, and the effect of quantity of water on plant growth depends on the quantity of sunlight. Similarly, the effect of octane rating on a motor’s performance depends on the compression ratio, and the effect of compression ratio on a motor’s performance depends on the octane rating of the gasoline.

In contrast to the OFAT design, DOE methods would call for a “factorial” design, which would include all four factor level combinations, as shown below.

By including that last data point, we can then know exactly what’s happening in the upper right hand area of the sample space, and we can also evaluate the interaction between A and B.

In the case of three factors—A, B, and C—the disadvantages of OFAT become even more apparent. A typical OFAT experiment is shown below. Here it’s obvious just how much of the operating envelope is being ignored.

In contrast to the OFAT design, the factorial design would include all eight factor level combinations, as shown below. This arrangement provides much better coverage of the operating envelope.

Our process has a large number of variables. Running a DOE will require a tremendous number of runs, won’t it?

In the previous example, we could see that each time we add a factor to a factorial design, the number of runs doubles. In other words, a factorial experiment with three factors requires eight runs, a factorial experiment with four factors requires 16 runs, an experiment with five factors requires 32 runs, and so on. The good news is that by using appropriate methods, the number of runs in a “full” factorial design can be cut by a factor of two, four, eight, etc. with relatively little loss of information. This is called a “fractional” factorial design.

Revisiting the case of three factors—A, B, and C—an appropriate half-fraction would look like the design shown below. Note that this design has the same number of runs (four) as the OFAT design shown above, but the fractional factorial design provides better coverage of the sample space.

To illustrate the real power of fractional factorial designs, consider the case of seven factors, for which the full factorial design would consist of 128 runs. If we’re interested only in main effects (i.e., ignoring the interactions), we can evaluate these seven factors in as few as eight runs, as shown below:

The OFAT design for seven factors, as shown below, would also have a total of eight runs, but there’s a key difference.

Even though both designs evaluate seven factors using eight runs, the fractional factorial design has the important advantage of being balanced. Notice that in the factorial design, for each factor there are four runs where that factor is at the high level, and four runs where that factor is at the low level. Comparing the means of those two groups will give us an estimate of the effect of that factor. In the OFAT design, however, each factor is run at the low level seven times and at the high level only once. That means that the effect of each factor is evaluated on the basis of a single data point, which is clearly less reliable.

Rather than using DOE, shouldn’t we be able to solve the problem in less time and with less expense using our expert knowledge?

DOE isn’t always the answer. Many challenging science and engineering problems have been solved by using trial and error coupled with expert knowledge. In fact, that’s how all problems were solved prior to the invention of DOE in the early 20th century. Keep in mind, however, that many of those problems took decades to solve.

If the problem is simple enough or the subject matter experts understand the process well enough, then trial-and-error may help you reach your goal quickly. The trial-and-error approach is risky, however. If, at any given point in time, the problem hasn’t been solved, then that means all the time and effort invested up to that point has largely been wasted. All you know at that point is that some design configurations work pretty well and some don’t, but you won’t have a clear idea of why.

For problems that real companies face related to product development, process development, and process improvement, the cost of engineering man-hours is insignificant compared to the cost of calendar time. With each week that passes with a new product not yet ready to release to market, you’re missing out on significant incremental revenue. With each week that passes with a product quality issue unresolved, you’re incurring significant costs that you could otherwise avoid. When facing complex problems, DOE methods allow you to explicitly model the relationships among the relevant variables and know exactly what direction to move in and why.

The trial-and-error approach, on the other hand, typically results in many moves being made in the wrong direction and precious time being wasted. Taking into account the cost of calendar time, a proper DOE approach is much less expensive than trial and error.

We tried DOE in the past with no success. Why didn’t it work?

With the wide variety of point-and-click software packages available for designing experiments, it seems all too easy to set up a DOE and analyze the resulting data. The fact is, in order to get reliable results from DOE, significant know-how and experience are required. Here are some of the most common reasons why a designed experiment doesn’t lead to the desired result:

Excessive measurement error
All too often, the measurement system adds so much random variation to the data that the DOE ends up yielding incorrect conclusions. Because of excessive noise in the data, you may conclude that significant factors are actually insignificant, or you may end up chasing noise, pursuing a design configuration that looked promising during the experiment but was really just a fluke.

Lack of understanding of the scientific and engineering principles in the system
DOE is a branch of statistics, and although some of the steps in the DOE process rely heavily on knowledge of statistical methods, other steps have little to do with statistics and have everything to do with understanding the physical and chemical phenomena occurring in the system. In-depth knowledge of science and engineering is essential for choosing the factors and levels used in the experiment and in interpreting the results of the statistical analysis.

Lack of capability in analyzing the data
All too often, the data that come out of a DOE are underutilized. One common but incorrect strategy is simply to choose the single best result from the experiment and use that as the starting point for the next experiment. That means that the decision regarding where to proceed is based on a single data point, and that all the information contained in the other data points is ignored. Another common strategy is to look at main effects plots for each of the factors and then set each factor to the level that yielded the best result. This ultimately means that interactions between the factors are ignored. Yet another incorrect strategy is to have the software generate a fancy statistical model without validating the underlying assumptions. To get the most out of the data from your experiment, you need to do proper statistical modeling.

Trying to run one big experiment
DOE is intended to be an iterative process. If you’re starting out with a large number of factors, then it’s likely that “you don’t know what you don’t know.” Rather than investing all your resources in one big experiment that’s supposed to answer all questions, you should run a series of smaller experiments, using the lessons learned from each experiment to guide you in the next experiment. This strategy will let you eliminate insignificant factors and identify unanticipated problems early on. It will also guide you gradually toward the most promising areas of the sample space so you can identify the “sweet spot” and study it in more detail.

Getting impatient
As mentioned above, DOE is an iterative process, and it requires rigor. With deadlines looming and customers demanding immediate results, there’s a strong temptation to take the results from the first experiment and then jump back into trial-and-error mode. From that point onward in the problem-solving process, you’re feeling your way around in the dark. You might get lucky and find a great solution, but you’ll more likely find a suboptimal solution, and in either case you won’t know how you got there.

What do I need to do to get started?

DOE is an entire process, and to follow that process successfully, you need three things: expert scientific and engineering knowledge, expert knowledge of statistical methods, and effective communication among those experts. Unfortunately, there are few individuals who, by themselves, have all the necessary knowledge. You probably already have the relevant science or engineering resources on staff, in which case what you lack is the knowledge of statistics. The temptation is to take a chemical engineer or other expert and send him to a weeklong course on DOE or Six Sigma. Experiments will be run, and sometimes the experiments will lead to a substantial improvement, but other times the end result will be team members standing around scratching their heads and wondering what went wrong.

The most effective solution is to hire a full-time industrial engineer or statistician. Preference shouldn’t be based on pure technical skill or academic credentials. Rather, the most important trait is the ability to communicate, as shown by experience in leading projects and working with other engineers, scientists, and managers. If your budget doesn’t allow for an additional full-time resource, then consider bringing in a temporary resource, whether that’s someone from a consulting firm, an academic institution, or your local Manufacturing Extension Partnership.


About The Author

Matt Treglia’s picture

Matt Treglia

Matt Treglia is a consultant with InnovaNet, an Atlanta-based provider of product commercialization, strategy development, and operations enhancement solutions. Treglia has been applying design of experiments and other statistical methods in various industries including building materials, consumer goods, medical devices, and polymers since 2001. He received a bachelor of science in mathematics and in statistics from the University of Tennessee, Knoxville, and a master of engineering in quality and reliability engineering from the University of Arizona. Treglia is a certified Six Sigma Black Belt.