Choosing the Right DOE Software

by H. Stith Bennett

Have you ever heard a quality colleague say something like "We did a DOE on it" or "Our people do DOEs all the time"? Every time I hear quality lingo used this way, I have contradictory feelings. I'm optimistic because experimentation seems to have achieved some measure of popular acceptance in the quality world. At the same time, this trendy use of a three-letter acronym promotes confused thinking regarding experiments.

Conventionally, DOE stands for design of experiments. So the phrases I mentioned previously become "We did a design of experiments on it" and "Our people do design of experiments all the time." I find such language confusing. The implication is that once an experiment is designed, it's done.

Can quality practitioners understand enough about the complex topic of designed experiments to use them effectively? Can computers help with experiments? Quality Digest's "DOE Software Buyers Guide" provides a way to begin sorting it all out. You will get the most out of it if you understand some of the frequently asked questions about experiments. The following are some basic questions to ponder.
What makes an experiment? Consider that all experiments have some rudiments in common. There are input variables, and there are outcome variables. The experimenter changes the input values and observes what, if anything, happens to the outcome values. All of us do this kind of thing every day. You change something and you see what happens. Usually, you change one thing at a time.

Although you can occasionally gain insights in this way, it is generally a poor way to gain knowledge. Nevertheless, in many business and industrial settings, this approach has historically been called "experimenting."

Designed experiments provide a much better way of going about things. In fact, you can think of experimental designs as ways to "fix up" everyday experiments so they succeed. When working with experimental designs, expect to pay in increased time and involvement to resolve your knowledge problems. But realize that the solution is designed experiments.
What kinds of experimental designs exist? There are more than can be counted or categorized conveniently. The names of some designs are derived from the settings where they were developed or extensively used. "Split plot" designs, for example, originated from agricultural research. Some design names refer to the people who invented or popularized them. "Plackett-Burman" and "Taguchi" are examples. Other design names refer to technical elements of the design itself. "Central composite" and "fractional factorial" are examples.

References to experimental designs are rooted in the ever-growing literature on experimentation. During the course of the last century, the universe of experimental design names has "just grown." Perhaps it comes as a disappointment that no clear naming convention exists to distinguish designs. For example, a Taguchi design may share some, but not all technical features with a composite design.

Furthermore, two names may be used to mean the same basic kind of design. For example, one person might refer to a design as a "second-order, three-level factorial," while another might refer to it as a "Box-Behnken" design.
What's the best design? Unless your grasp of matrix algebra is good and you have done extensive reading of the literature of experiments and linear models, don't pretend you know how to compare the merits of all designs. Instead, use your time wisely. Create a statement of your experimental motive. Then seek help from someone who is truly knowledgeable, use appropriate computer software to guide you to a design or use the simplest design you know how to use. Then forge ahead against all odds and do the experiment.
Why experiment? Instead of tackling experimentation by sorting through all the possible design types, start by figuring out your own experimental motive. Why do you want to experiment? What kind of knowledge do you want to have when the experiment ends?

Two examples of the many possible experimental motives are screening and empirical modeling. A screening motive asks this question: "Out of all these inputs, which ones affect the outcomes?" Screening motivates the first stages of inquiries to sift through many input variables of unknown importance. An empirical modeling motive asks another question: "If the inputs are set to certain values, what will the outcome values be?" Empirical modeling seeks input values that result in desirable outcome values.

Recent DOE converts often fail to understand that motives for experimenting can differ. Resist the temptation to identify emotionally with some favorite category of allegedly superior designs.

Understanding your motive for experimenting provides you with the information you'll need to select an appropriate design. You will come up with answers to questions like these: "How many input and outcome variables must be handled? How much time and money can you invest? What higher-order relationships need to be resolved? How are experiments performed?"

As DOE fever has spread in recent years, many quality practitioners have failed to do correctly the experiments they have spent so much time learning to design correctly. Botched data acquisition, nonexistent validation of required input-level values, screwed-up timing, inattention to changing environmental variables and imposed human bias can limit the possibility that even the best experimental design can advance understanding.
Why is it nonsense to say, "We did a DOE"? Somehow, the three letters that stand for the design of experiments have been confused with the doing of experiments. Despite the implications of popular lingo, designing an experiment is not doing one. Designing an experiment is just the first of many steps in gaining knowledge through experimentation. And the correct design is no more important than correct planning, correct execution and correct analysis of the experimental data.

Consider that the results of an experiment with the "wrong" design that is nevertheless well-executed have more value than the results of an experiment with the "right" design where the execution is a failure. Of course, you really want it all: good design, planning, execution and analysis.
What should DOE software offer? Consider your motives, and think of what you want to do. You will want to plan for the successful performance of experiments and actually perform experiments. You will want to acquire and analyze experimental data, arrive at appropriate findings based on the analysis and com- municate your findings in reports. And, most importantly, you will want to apply your findings to your process in a way that gets results.

Look for features that help you do what you want to do. Some DOE software merely helps you select a design. Most recent DOE software can also analyze experimental data resulting from the designs it provides. A new trend is to provide users with expert advice while interpreting any data analysis performed. However, you probably will be on your own when it comes to the nuts and bolts of planning and performing your experiment.

Usually, you also must initiate the creation of the data set that the software analyzes, although the software may provide a form with a set of database fields or a link to a third-party spreadsheet so it can locate the experimental data you enter for later analysis. Most DOE software has reporting capabilities or is linked to a database management system that makes reports. Don't expect a lot of help from DOE software in applying your findings.

My best advice to DOE software shoppers is to create a written set of requirements based on your motives for experimenting. Avoid listing software features. Rather, try to list what you need accomplished. If you have a hard time imagining everything, flowchart the process with which you're going to experiment. What are the possible input and output variables? What reports do you need? How will experimental results be implemented? What kinds of training and support do you need? Prioritize your requirements.

Be realistic. Recognize the gaps between what DOE software can do and what your experimental motive requires to be done. Figure out how you will fill those gaps before you make a purchase.

Visualize your experimental setting. Can you see exactly what tasks the software performs? Can you see the tasks you (or other people) perform? Is there anything wrong or missing from this picture? If this process brings up new questions, seek the answers.

About the author
H. Stith Bennett has taught at the University of Illinois, the University of Washington and the University of Missouri. In 1980, he began Colorado Observations, a general research and statistical consulting firm. In 1987, he co-founded SynchroStat Systems Corp., a software development company specializing in real-time statistical applications for industry and science.