The Bickerson’s Battle Over Alpha

Meet Betty and Bart Bickerson, husband-and-wife quality analysts who work at different companies. Betty and Bart argue about everything. They argue whether grey is a color. They argue whether tomato is a fruit. They argue whether the chicken came before the egg, and whether the egg tastes better fried, scrambled, or poached. But their relationship didn’t get really rocky until they started to argue about what alpha level, also called the “significance level,” to use for a hypothesis test.

Note: The alpha level is the criterion against which you compare the p-value to determine whether a difference is statistically significant. The alpha level option for the two-sample t test in the Minitab Assistant is shown below:

Listen in as Betty and Bart squabble over what alpha level to use when comparing the mean before and after a process change.

Bart: “Well, Betty, I think I’m going to raise the alpha level for the test. Kick it up a notch from 0.05 to 0.10.”

Betty: “That’s funny. If anything, I was going to lower it a notch—down to 0.01. Why would you ever want to raise the level of risk?”

Bart: “I want to boost the power of the test to detect a difference. If I use a higher alpha, I can be more certain I’m going to find a significant difference between the process means, if it’s really there.”

Betty: “But if you raise alpha, you’re also going to be less certain that any significant shift in the process mean that you find is really true and not just a statistical fluke—a random error. You’ll increase the risk of false positives.”

Bart: “But by lowering alpha, you’ll increase the risk of false negatives. The last thing I want is to miss a possible effect on the process mean. Even after I raise alpha to 0.1, the risk of falsely finding a significant result will only be 10 percent.”

Betty: “Well, Bart, I don’t want to mistakenly conclude there’s a significant process change when there’s really not. That’s worse in my book.” (Straightens her posture and adjusts the “Quality Counts” button on her lapel. “And if I find a significant difference, I’ll be 99-percent confident in my results. You’ll only be 90-percent confident.”

Bart: “You don’t need to be sooo worried about reporting a significant effect that’s not there.”

Betty: “And you don’t need to be sooooooo paranoid about missing a significant effect.”

Bart: “I’m not paranoid. You’re paranoid.”

Betty: “Well, at least I don’t snore like a leaf blower stuck in full throttle.”

Bart: “And at least I don’t crack my toes in bed.”

Can statistical therapy save this relationship?

It was time for Betty and Bart to see a statistical therapist. After years of data analysis, Dr. Sigma Freud fully understood what constituted normality—and what didn’t. Freud listened patiently to Betty and Bart spar over the alpha level. After they had temporarily run out of lung capacity, she turned to Betty.

Dr. Freud: “Your fear of incorrectly concluding that a result is significant, when it’s really not, is completely justified. In fact, we have a special name for that faux pas in statistics: It’s called a Type 1 error.”

Betty smiles and tries not to look smug.

Dr. Freud: And just as you say, Betty, you can reduce the chance of making a Type I error by lowering the alpha level for a hypothesis test.”

Freud turns to Bart.

Dr. Freud: “And Bart, your concern is completely valid as well. If a hypothesis test does not reveal a significant difference, but a difference does indeed exist, we call that a Type II error. And one way to help guard against it is to raise the alpha level.”

Was it Betty’s imagination, or was Bart’s normally concave chest starting to puff up a bit?

Dr. Freud: “So both of your concerns are valid. This push-pull dynamic of risk is a natural part of any healthy statistical relationship. What we need do is to find a way to balance those competing risks.”
Betty: “How can do we do that?”

Dr. Freud: “You know, what about just going with the default alpha level of 0.05? That’s what most couples would do. You’d be 95-percent confident that any shift in the process mean that you find is real.”

Betty: “Hmmm. Sometimes I do wish we’d never clicked the arrow to see the options for alpha. Life was simpler.”

Bart shakes his head.

Bart: “Too many things have been said. We can’t go back now.”

Why two rights don’t make a wrong

Dr. Freud: “OK. Let’s take a step back and delve a little deeper. Why don’t you both tell me more about the projects you’re working on. Betty, let’s start with you.”

Betty: “I work for a medical device company. We’ve developed a new implant device for patients. We're excited because it might really improve patient outcomes. But it’s more costly than the current device. Before we put it into production, we want to be doubly sure that the difference in patient outcomes is really there.”

Dr. Freud: “So if you find a difference in outcomes—specifically, an improved outcome with the new device—and that difference really doesn’t exist, the consequences could be very severe.”

Betty: “Absolutely. If the new device doesn’t improve outcomes, and we don’t realize it, we’d waste a lot of money producing it. Patients would also have to pay more for something that didn’t really offer a real improvement over the current implant device.”

Dr. Freud: "Well Betty, in your case, the consequences of a Type I error are more severe than a Type II error. So you’re doing exactly what you should be by lowering the alpha level. Now, Bart, what about you?”

Bart: “I work in an automotive plant. One of our parts suppliers has been raising prices. We want to switch to another supplier that offers the same part at a lower price. But before we do that, we need to compare key dimensions of the part to make sure there’s no difference. We don’t want to sacrifice quality for reduced cost.”

Dr. Freud: “So if you don't find a difference in the dimensions in the part between the two suppliers, and a difference really does exist, the consequences could be very severe.”

Bart: “Absolutely. If the dimensions are off, and we don’t realize it, we could lose a lot of money with increased scrap rates. Overall product quality could also drop, and we might lose customers.”

Dr. Freud: “Well Bart, in your case, the consequences of a Type II error seem to be more severe than a Type I error. So you’re doing exactly what you should be by raising the alpha level.”

Betty/Bart (together): “You mean we’re both right?!”

Dr. Freud: “Absolutely! Now how about a little hug?”

The Bickersons embrace, overjoyed at finally understanding the alpha level.

Betty: “I’m so glad I suggested coming here!”

Bart: “Me, too! But, actually, didn't I suggest coming here?”

Betty: “No, Bart, I’m sure it was my idea.”

Bart: “Au contraire, Betty...”

Moral of the story

Most of the time, you can leave the default alpha level (0.05) alone. Unless you’re in a situation like Betty or Bart. If you need statistical therapy, check out the Minitab Assistant. It’s like having a statistical therapist built right into the analysis itself.

If you'd prefer a real, live statistical therapist to help you design or analyze your quality improvement project, check out the mentoring service offered by Minitab trainers.

About The Author