Featured Product
This Week in Quality Digest Live
Six Sigma Features
Ken Maynard
How to create a continuous improvement culture in any organization
Jody Muelaner
A standard set of graphical methods for improving quality
Dirk Dusharme @ Quality Digest
Making better use of taxpayer dollars
Ryan E. Day
Case studies in saving taxpayer dollars with lean Six Sigma
Taran March @ Quality Digest
Lean Six Sigma has arrived on a few campuses. Will more follow?

More Features

Six Sigma News
Floor symbols and decals create a SMART floor environment, adding visual organization to any environment
A guide for practitioners and managers
Making lean Six Sigma easier and adaptable to current workplaces
Gain visibility into real-time quality data to improve manufacturing process efficiency, quality, and profits
Makes it faster and easier to find and return tools to their proper places
Version 3.1 increases flexibility and ease of use with expanded data formatting features
Provides accurate visual representations of the plan-do-study-act cycle
SQCpack and GAGEpack offer a comprehensive approach to improving product quality and consistency
Customized visual dashboards by Visual Workplace help measure performance

More News

Steve Moore

Six Sigma

Feel-Good Data Analysis Won't Save Firefighters’ Lives

“Wishin’ and hopin’ and thinkin’ and prayin’...”

Published: Wednesday, August 4, 2010 - 06:00

An article titled “Sharp Drop in Firefighter Fatalities in 2009” appeared in a safety trade journal recently. For the first time in three years, it said, on-the-job firefighter fatalities dropped below 100. The article went on to say that the 82 firefighter deaths in 2009 were substantially below the 10-year average of 98 and down from 105 in 2005.

What prompted me to write this article was the following quote: “While a drop over one year isn’t enough to show a trend… [w]e are hopeful that we will continue to see fewer and fewer firefighter fatalities over the next 10 years.” I was reminded of the old song by Dusty Springfield, “Wishin’ and Hopin’,” in which a young woman sadly admits she has to do more than “wishin’ and hopin’ and thinkin’ and prayin’” to be his. She has to do something differently.

The article referenced a National Fire Protection Association (NFPA) report, and I just had to pull it up and look at the data myself. What I found was 45 pages of “this number is greater than that number” and “that number is less than average.” Every change from year to year was treated as if it was due to special cause variation. Lots of bar charts and pie charts categorized the firefighters’ ages, types of duty, cause of injury, and nature of injury. Good information, perhaps, but I got the feeling the report was written to impress the reader with volume and charts—the wrong charts. Although the report did not contain much data from before 2009, it did contain data for firefighter deaths from 1979–2009 in the form of a bar chart. Unfortunately, data analysis stopped there.

As Davis Balestracci and Donald Wheeler often say, “Plot the data!”  So I did. The result was the process behavior chart seen in figure 1:

Figure 1: Process behavior chart

Connecting the dots

Data that are in time order have an extra dimension that must be taken advantage of. Time-ordered data allow us to “hear” the voice of the process.  We must learn to “listen” with the aid of process behavior charts. The chart clearly shows us that firefighter fatalities have indeed decreased, but not in 2009.  The so-called “sharp drop” from 2008 to 2009 was due to common cause variation.

However, something significant did happen to reduce firefighter fatalities during the period 1991–1994. The NFPA could benefit by investigating what changes occurred to reduce firefighter deaths from an average of 123 per year to 99. Were there changes in technology, training, health screening? What about 1992 and 1993? These two years should certainly raise some questions. Whatever caused the decrease in firefighter fatalities, it must be identified and continued or enhanced. Otherwise, we can expect that somewhere between 77 and 121 firefighters will die on the job in 2010 and beyond.

A complete and proper analysis of the data is beyond the scope of this article.  However, I would like to show one more step in the analysis to help make a point. The NFPA report analyzed the data in terms of fatalities per 100,000 structure fires (see figure 2). A graph was shown with the caption, “While the number of structure fires and deaths at structure fires has dropped, the rate of firefighter deaths at structure fires has not followed the same pattern.”

Another process behavior chart shows that this is not the case. The rate of firefighter deaths decreased twice: once around 1983–1984 and again, more significantly, during 2000–2001. Both these signals of improvement have been missed by the NFPA. Knowledge of what caused these shifts may help save lives in the future.

Figure 2: Firefighter deaths per 100,000 structure fires

Ignoring the forest for the trees

Although I have tended to criticize the NPFA report and its lack of proper statistical thinking to gain proper insight for performance improvement, there is one nugget of information that warrants further investigation. According to the report, about 40 percent of firefighter fatalities are sudden cardiac deaths (e.g., heart attacks or strokes), by far the largest category, which underscores “the need for wellness-fitness programs and health screenings for firefighters across the nation.” The NFPA also reports that 44.7 percent of 2,000 firefighters tested in 2005 were found to be obese. Of 1,650 firefighters tested in 2008, 47.5 percent were determined to have high overall coronary risk ratings.

These data may not be scientifically valid—due to sampling protocol—but they are strong evidence that our nation’s firefighters may not be fit for such strenuous duty. Obviously, fitness should be a focus area for the prevention of firefighter deaths on the job. The second largest category of firefighter deaths is vehicle accidents while responding to or returning from alarms.

The only reason to collect data is to turn the information into knowledge (i.e., insight) as a basis for action. We are constantly bombarded by the media with statistics that are misused and abused to tell a story that the author wants to tell. As a result, the insights needed to gain improvement often are lost. In this case, the impression was given that firefighter deaths are trending downward. They are not. Feel-good analysis, coupled with “wishin’ and hopin’” that better times are ahead, will get us nowhere. Our nation’s firefighters deserve better.

Discuss

About The Author

Steve Moore’s picture

Steve Moore

After 47 years, Steve Moore is retired from the pulp and paper industry. He is a graduate of North Carolina State University with a pulp and paper degree, and holds a master's degree from the Institute of Paper Chemistry in Appleton, Wisconsin. He has held various research and development, technical, engineering, and manufacturing positions in the paper industry. He has been a student, teacher, and practitioner of statistical methods applied to real-world processes for the past 35 years.

Comments

Another theory of cause

I recently saw results of a study that ranked firefighting as one of the most "mentally" stressful jobs - again a condition that could be contributing to cardiac problems along with or instead of physical shape. I liked your article and the use of the data to strive for improvement instead of just being used to make it "sound" like things are getting better for unstated reasons.

Operational Definitions

Nice article. It amazes me how the media continues to get in wrong (numerical naivete).
One very important thing that is often over-looked when analyzing time series data is the need of Operational Definitions (how the count or measurement is defined). In fact, according to Deming, Shewhart considered this to be more important than his invention of the control chart. I know with the number of adminstrative metrics that are used to compare data from different months, departments and plant sites etc, the importance of OD can't be emphasized enough in my view. For example, if a fire fighter dies from a stroke or heart attach 3 days after fighting a fire, is that included as a job related fatality? I often wonder when I see evidence of improvement, whether the measurement process has changed.

Rich