Properties of Probability Models, Part 2

Story update 9/8/2015: There was an error in the data set for columns "Skew" and "Kurt" in figure 7. The error has been corrected.

Clear thinking and simplicity of analysis require concise, clear, and correct notions about probability models and how to use them. Last month in part one we looked at the properties of the Weibull probability models and discovered that some ideas about skewed distributions are incorrect. Here we shall examine the basic properties of the family of Gamma models.

How would you characterize a skewed distribution? When asked this question most will answer, "A skewed distribution is one that has a heavy, elongated tail." This idea is expressed by saying that a distribution becomes more heavy-tailed as its skewness and kurtosis increase. Last month, for Weibull models at least, we discovered that as the tail was elongated it grew lighter, not heavier. Does this happen with other families of probability models? Here we consider the Gamma models.

The gamma family of distributions

Gamma distributions are widely used in all areas of statistics and are found in most statistical software. Since software facilitates our use of the Gamma models, the following formulas are given in the interest of clarity. Gamma models depend upon two parameters, once again denoted by alpha, α, and beta, β. The probability density function for the Gamma family has the form:

The alpha parameter determines the shape of the Gamma model. When the value for alpha is 1.00 or less, the Gamma distributions will be J-shaped. As the value for alpha increases above 1.00, the Gamma distributions become mound-shaped, and as the value for alpha gets large, the Gammas approach the normal distribution. Since we consider these distributions in standardized form, the value for the beta parameter will not affect any of the following results. Six standardized Gamma distributions are shown in figure 1.

Figure 1: Six standardized gamma distributions

So what is changing as you select different Gamma probability models? To answer this question, figure 2 considers 19 different Gamma models. For each model we have the skewness and kurtosis, the areas within fixed-width central intervals (encompassing one, two, and three standard deviations on either side of the mean), and the z-score for the 99.9th percentile of the model.

The z-scores in the last column of figure 2 would seem to validate the idea that increasing skewness corresponds to elongated tails. As the skewness gets larger, the z-score for the most extreme part per thousand also increases. This may be seen in figure 3, which plots the skewness vs. the z-scores for the most extreme part per thousand. So skewness is directly related to elongation, as is commonly thought. But what about the weight of the tails?

Figure 2: Characteristics for various gamma models

Figure 3: Skewness and elongation for gamma models

Figure 4 plots the areas for the fixed-width central intervals against the skewness of models from figure 2. The bottom curve of figure 4 (k = 1) shows that the areas found within one standard deviation of the mean of a Gamma distribution increase with increasing skewness. Since the tails of a probability model are traditionally defined as those regions that are more than one standard deviation away from the mean, the bottom curve of figure 4 shows us that the areas in the tails must decrease with increasing skewness. This contradicts the common notion about skewness and a heavy tail.

Figure 4: How the coverages vary with skewness for gamma distributions

So while the infinitesimal areas under the extreme tails will move further away from the mean with increasing skewness, the classically defined tails do not get heavier. Rather, they actually get much lighter with increasing skewness. To move the outer few parts per thousand further away from the mean, you have to compensate by moving a much larger percentage closer to the mean. This compensation is unavoidable and inevitable. To stretch the long tail, you have to pack an ever-increasing proportion into the center of the distribution!

Figure 5: How the tails get lighter with skewness for Gamma distributions

So while skewness is associated with one tail being elongated, that elongation does not result in a heavier tail, but rather in a lighter tail. Increasing skewness is rather like squeezing toothpaste up to the top of the tube: While concentrating the bulk at one end, little bits get left behind and are squeezed down toward the other end. As these little bits become more isolated from the bulk, the "tail" becomes elongated.

However, once again, there are a couple of surprises about this whole process. The first of these is the middle curve of figure 4 (k = 2), which shows the areas within the fixed-width, two-standard-deviation central intervals. The flatness of this curve shows that the areas within two standard deviations of the mean of a Gamma stay around 95 percent to 96 percent regardless of the skewness.

In statistics classes students are taught that having approximately 95 percent within two standard deviations of the mean is a property of the normal distribution. Last month we found that this was a property of the family of Weibull models. Here we see that this property also applies to the Gamma distributions. Beginning with the mound-shaped Gammas and continuing through the J-shaped Gammas, there will be approximately 95 percent to 96 percent within two standard deviations of the mean.

The second unexpected characteristic of the Gammas is seen in the top curve of figure 4 (k = 3), which shows the areas within the fixed-width, three-standard-deviation central intervals. While the area within three standard deviations of the mean does drop slightly at first, it stabilizes for the J-shaped Gammas at about 97.5 percent. This means that a fixed-width, three-standard-deviation central interval for a Gamma distribution will always contain at least 97.5 percent of that distribution.

Figure 6: What gamma distributions have in common

So if you think your data are modeled by a Gamma distribution, then even without any specific knowledge as to which of the Gamma distributions is appropriate, you can safely say that 97.5 percent or more will fall within three standard deviations of the mean, and that approximately 95 percent or more will fall within two standard deviations of the mean. Fitting a particular Gamma probability model to your data will not change either of these statements to any practical extent.

For many purposes these two results will be all you need to know about your Gamma model. Without ever actually fitting a Gamma probability model to your data, you can filter out either 95 percent or 98 percent of the probable noise using generic, fixed-width central intervals.

What gets stretched?

If the tail gets both elongated and thinner at the same time, something has to get stretched. To visualize how skewness works for Gamma models we can compare the widths of various fixed-coverage central intervals. These fixed-coverage central intervals will be symmetrical intervals of the form:

MEAN(X) ± Z SD(X)

While this looks like the formula for the earlier fixed-width intervals, the difference is in what we are holding constant and what we are comparing. When we hold the widths fixed, we compare the areas covered by the intervals. When we hold the coverages fixed, we compare the widths of the intervals. These widths are characterized by the z-scores in figure 7. For example, a Gamma model with an alpha parameter of 1.25 will have 92 percent of its area within 1.53 standard deviations of the mean, and it will have 99 percent of its area within 3.49 standard deviations of the mean.

Figure 7: Widths of fixed-coverage central intervals for gamma models

Figure 8 shows the values in each column of figure 7 plotted against skewness. The bottom curve shows that the middle 92 percent of a Gamma will shrink with increasing skewness. The 95-percent fixed-coverage intervals are remarkably stable until the increasing mass near the mean eventually begins to pull this curve down. The 97.5-percent fixed-coverage intervals initially grow until they plateau near three standard deviations.

Figure 8: Widths of fixed-coverage central intervals for Gamma models

The spread of the top three curves shows that for the Gamma models, it is primarily the outermost 2 percent that gets stretched into the extreme upper tail. While 920 parts per thousand are moving toward the mean, and while another 60 parts per thousand get slightly shifted outward and then stabilize, it is primarily the outer 20 parts per thousand that bear the brunt of the stretching and elongation that goes with increasing skewness.

The benefits of fitting a gamma distribution

So what do you gain by fitting a Gamma model to your data? The value for the alpha parameter may be estimated from the average and standard deviation statistics, and this estimate will, in turn, determine the shape of the specific Gamma model you fit to your data. Since these statistics will be more dependent upon the middle 95 percent of the data than the outer one percent or less, you will end up primarily using the middle portion of the data to choose a Gamma model. Since the tails of a Gamma model become lighter with increasing skewness, you will end up making a much stronger statement about how much of the area is within one standard deviation of the mean than about the size of the elongated tail. Fitting a Gamma distribution is not so much about the tails as it is about how much of the model is found within one standard deviation of the mean. So, while we generally think of fitting a model as matching the elongated tail of a histogram, the reality is quite different.

Once you have a specific Gamma model, you can then use the model to extrapolate out into the extreme tail (where you are unlikely to have any data) to compute critical values that correspond to infinitesimal areas under the curve. However, as may be seen in figures 3 and 8, even small errors in estimating the parameter alpha can have a large impact upon the critical values computed for the infinitesimal areas under the extreme tail of your Gamma model. As a result, the critical values you compute for the upper one or two percent of your Gamma model will have virtually no contact with reality. Such computations will always be more of an artifact of the model used than a characteristic of either the data or the process that produced the data.

To illustrate this point, I generated 5,000 data sets of 100 values each using an exponential distribution (which is a Gamma with an alpha parameter of 1.000). For each data set I estimated the value of alpha. These estimates ranged from 0.495 to 2.103. From figure 3 we can see that this range of values for alpha will result in Gamma models that have their most extreme part per thousand anywhere in the range from five SD to seven SD above the mean. Thus, the uncertainty in your estimate of the alpha parameter will create large uncertainties in the location of the infinitesimal areas under the extreme tail. Consequently, any extreme tail critical values you compute will be more of an artifact of your model than a characteristic of your data.

Industrial data analysis

What impact does all this have on how we analyze data? It turns out that there are two distinctly different approaches to data analysis. For clarity call these the statistical approach and Shewhart's approach.

The statistical approach uses fixed-coverage intervals for the analysis of experimental data. In some cases these fixed-coverage intervals are not centered on the mean, but rather involve fixed coverages for the tail areas, but this is still analogous to the fixed-coverage central intervals used above. Fixed coverages are used because experiments are designed and conducted to detect specific signals, and we want the analysis to detect these signals despite the noise present in the data. By using fixed coverages, statisticians can fine-tune just how much of the noise is being filtered out. This fine-tuning is important because additional data are not generally going to be available, and we need to get the most out of the limited amount of experimental data. Thus, the complexity and cost of most experiments will justify a fair amount of complexity in the analysis. Moreover, to avoid missing real signals within the experimental data, it is traditional to filter out only 95 percent of the probable noise.

Shewhart's approach was created for the continuing analysis of observational data that are the byproduct of operations. To this end Shewhart used a fixed-width interval rather than a fixed-coverage interval. His argument was that we will never have enough data to ever fully specify a particular probability model for the original data. Moreover, since additional data will typically be available, we do not need to fine-tune our analysis—the exact value of the coverage is no longer critical. As long as the analysis is reasonably conservative, it will allow us to find those signals that are large enough to be of economic importance without getting too many false alarms. So, for the real-time analysis of observational data, Shewhart chose to use a fixed-width, three-sigma central interval. As we have seen, such an interval will routinely filter upwards of 98 percent of the probable noise.

Figure 9: How three-sigma limits work with Gamma distributions

What we have discovered here is that Shewhart's simple, generic, three-sigma limits will provide a conservative analysis for any and every data set that might logically be considered to be modeled by a Gamma distribution. Last month we discovered that Shewhart's simple, generic, three-sigma limits also provide a conservative analysis for any and every data set that might logically be considered to be modeled by a Weibull distribution. This is why finding exact critical values for a specific probability model is not a prerequisite for using a process behavior chart. Once you filter out approximately 98 percent or more of the probable noise, anything left over is a potential signal.

About The Author