n my February 1996 Quality Digest column I discussed an article out of USA Today. Since that article provides a great example of how we need to filter out the noise whenever we attempt to interpret data, I have updated it for my column today. “Teen Use Turns Upward” read the headline for a graph appearing in USA Today on June 21, 1994. The data in the graph were attributed to the Institute for Social Research at the University of Michigan and were labeled as the “percentage of high-school seniors who smoke daily.”
The portion of this graph covering the 10 years from 1984 to 1993 is shown in figure 1.
Each point in figure 1 is the value found in an annual survey. The 1993 value of 19.0 percent was higher than the 1992 value of 17.3 percent. This was interpreted to mean that more teenagers are using tobacco now than in the past. But are they?
Before we can make sense of numbers such as these we need to know something about the limitations of data from surveys. First of all, values such as these are subject to variation. Two identical surveys carried out at the same time will rarely yield identical results. Among the sources of variation for survey data are differences in who is interviewed, differences in how they are interviewed, differences in how they respond, and differences in how their responses are reported. Second, when a survey is used from year to year, there are also the problems of different personnel being used to conduct the survey, and differences in how the questions are perceived in different years.
Therefore, no matter what is being measured, and no matter how carefully it is measured, the statistics will always vary. There will always be some variation in survey results from year to year. Even if nothing is changing, we can expect the value to go up about half the time, and we can also expect the value to go down about half the time. (Look at the past nine years—these values were greater than the previous value four times, and less than the previous value five times.)
So how do we ever detect a change using survey data? If we interpret each and every change in the percentage who smoke daily as a year-to-year difference, how do we know that we are not being misled by the variation from survey to survey? If we admit that there is variation between surveys, how then do we ever know when there has been a change from one year to another?
The answer is that we must first filter out the routine variation between surveys, and then look for year-to-year differences. The simplest way to do this is with a process behavior chart.
We begin with the yearly values. For the data on high-school students who smoked daily, the annual percentages reported for 1984 through 1993 were, respectively:
The average of these 10 values is 18.64 percent. This average is used as a central line, and the 10 values are plotted as a time series as shown in figure 2. This graph is the beginning of the chart for individual values (also known as an X-chart).
Since the variation between one year’s value and the next will always include the routine variation between surveys, we use the year-to-year variation as our guide to how much uncertainty is inherent in the reported results. These year-to-year changes are measured by the differences between successive values (these differences are called moving ranges). The nine moving ranges for these data are:
The average moving range is 0.778 percent. We use this average moving range to compute limits for both the X-chart and the moving range chart (mR-chart). The limits for the X-chart are commonly known as natural process limits. They are placed symmetrically on either side of the central line. The distance between the central line and these limits is found by multiplying the average moving range by 2.660. This value of 2.660 is a constant that converts the raw statistic into the appropriate measure of dispersion. In this case, the natural process limits are computed to be 16.57 percent to 20.71 percent.
These limits make allowance for the routine variation. Before any single value can be said to represent a change in the use of tobacco by teenagers, it will have to either exceed the upper limit or fall below the lower limit. Since none of these values fall outside these limits, any statement about changes in the percentage of teens who smoke is questionable.
But wait—the change between the last two values, where the percentage jumped from 17.3 percent to 19.0 percent, is the biggest change during the past 10 years. Surely this should mean something. To see if this is the case, we place the moving ranges on a chart. The average moving range of 0.778 will be the central line for this chart, and the upper limit will be found by multiplying the average moving range by 3.27. This results in an upper range limit of 2.54 percent.
The last value on this moving range chart shows the “jump” between 1992 and 1993. This moving range of 1.7 percent does not fall above the upper limit of the moving range chart. Thus, once again, the “jump” from 17.3 percent to 19.0 percent does not qualify as a clear-cut signal.
So, what can we say about the percentage of teenagers who smoke daily? Just this: The data presented show no evidence that the percentage of teenagers who smoke has increased. Neither is there any evidence that this percentage has decreased during the past 10 years. The only headline for these data that has any integrity is “No Change in Teen Use of Tobacco.” Anything else is just propaganda.
But what about teens who smoke while driving? The point of this column is that you should be careful about believing the headlines. However, since driving while distracted is currently a hot topic for news, I give you the headline from the Dec. 14, 2011, USA Today, “19 states see jump in traffic fatalities.” The subtitle was, “Masked by national decline for fifth year.” The lead paragraph in this story is, “Even as U.S. traffic fatalities dropped in 2010 for the fifth straight year, more than a third of the states saw their deaths increase.” Presumably 31 states reported a decline in traffic fatalities, but evidently the editors wanted to spin the story by talking about the 19 states where traffic fatalities went up. Of course, the implicit assumption behind this headline is that every change is a signal. Unfortunately, this is simply not true. The first mistake of data analysis is to interpret noise as a signal. The second mistake is to fail to detect a signal that is present. When you interpret every change as a signal, then you will always be distracted by the noise that is present in all data, and you will become fair game for the propaganda artists. The result of all this is that the more you listen to the news these days, the less you know.
So how do you avoid being persuaded by propaganda? The first principle of data analysis is that no data have any meaning apart from their context. This automatically precludes interpreting isolated values or even limited comparisons between pairs of points. The second principle of data analysis is that while all data contain noise, only some data contain signals. If you do not know how to separate the probable noise from the potential signals you are susceptible to being misled by the noise in the data. Others may use data to mislead you—or you may even mislead yourself. While I have written a whole book on these two principles, Understanding Variation, the Key to Managing Chaos (SPC Press, 2000), the short and simple answer is that process behavior charts are the simplest way to see the data in context and filter out the noise.
By the way, did you read the article about how the trade deficit soared last April? Oh, well, that’s another story—or is it?