Given a Set of Numbers, 25% Will Be the Bottom Quartile... or Top Quartile

Time to get out the Ouija boards

Published: Thursday, October 17, 2013 - 12:06

As many of you know, I hate bar graphs. They are ubiquitous, and most of them are worthless. I'll make maybe two exceptions: 1) a Pareto analysis; 2) a comparative set of stratified histograms disaggregating a stable period of performance (a Pareto analysis proxy for continuous data). Displaying the latter as bars is one option; another—and my preferred—is displaying run or control charts on the same page and same scale.

About 15 years ago, I picked up my morning paper and saw an article, with the accompanying graph below (see my last column), rating the 20 health systems in my metropolitan community on the question, “Would you recommend your clinic to adult friends or family members?” I happened to work at Clinic 19 (maybe not the best service, but excellent care).

Click here for larger image.

By all means, let’s give the public data upon which to make a decision. But, based on this graph, what would you would decide? As for me, I have absolutely no idea.

As much as I hate plain bar graphs, worthless stacked bar graphs like this one make my blood boil. Look at all that wasted ink. Here’s a cost-cutting suggestion: Ask your executives to consider taking the color printer out of the executive wing—those colored ink cartridges are expensive.

When I arrived at work that morning, I was greeted by (surprise!), “Davis, we’d like you to take a look at that data!” That is the only time the VP of marketing had ever shown any interest in my work. I was able to obtain these data after a few phone calls.

Off to the Milky Way? Or data sanity?

It was a classic customer satisfaction survey based on a 1 to 4 Lichert scale: 1 = Definitely no, 2 = Probably no, 3 = Probably yes, 4 = Definitely yes. Note that the table combines the Definitely no and Probably no responses. I had no idea how the sample was chosen, guessing that they kept sending surveys out (randomly) until 250 responses were received (not so random, but that’s another story).

The figure below shows the tool of choice and its use for analyzing such data—they are placed underneath a Ouija board:

Of course, I’m pulling your leg, but for all intents and purposes, that’s pretty much what’s going on. That said, a similar ideomotor effect that propels the planchette around a ouija board is taking place with the graph: People see what they want to see.

A simple, counterintuitive, more productive statistical alternative

Rather than calculate the usual weighted average, then ranking, which pretty much results in a meeting like the above, I decided to do two analyses via a p-chart analysis of means. The first used “Definitely yes,” data, and the second used the combined “Definitely no” and “Probably no” data.

Because the sample sizes for the 20 systems are so close, I also decided to use the average sample size in the calculation—i.e., 245. Rule of thumb for a p-chart: As long as an individual clinic’s sample size is within plus or minus 25 percent of this number (~185–305), it’s a good approximation to start.

Chart 1: Definitely yes
As you see, the overall average of the 20 systems is 57.4 percent people answering “Definitely yes” (known in marketing jargon as “top box”). Given that and using 245 responses per clinic, one must answer the question for each clinic. Is it possible that their result is just random statistical variation from 57.4 percent, i.e., “innocent until proven guilty?” The analysis and chart are shown below:

57.4 + [3 x Square Root (57.4 x (100 – 57.4) / 245)] = [48.0% – 66.9%] Common cause range

Two systems (11 and 12) are truly above average (good), and two systems (7 and 15) are truly below average (bad).

Chart 2: Definitely no and Probably no
In a similar manner, the system average is 7.3 percent of the people answering “Definitely no” or “Probably no,” resulting in the analysis:

7.3 + [3 x Square Root (7.3 x (100-7.3) / 245)] = [2.3% - 12.3%] Common cause range

One system (15) is truly above average (bad—as they were in the previous analysis), and no system is below average.

Is it clearer now whom you might choose?

Quartilemania
Another obsession with data like these is to rank the 20 systems via quartiles. So, for the best and worst possible responses, this results in:

Definitely yes:
Quartile 1 (alleged best): 1, 11, 12, 16, 18

Quartile 2 (allegedly pretty good—still above average): 4, 5, 13, 17, 20

Quartile 3 (uh-oh, below average): 3, 6, 8, 9, 14

Quartile 4 (alleged absolute worst): 2, 7, 10, 15, 19

Probably not and Definitely not:
Quartile 1 (alleged absolute worst): 3, 7, 8, 15, 19

Quartile 2 (uh-oh, below average): 2, 5, 6, 9, 14

Quartile 3 (Whew! Better than average) 4, 10, 11, 16, 18

Quartile 4 (alleged best): 1, 12, 13, 17, 20

Yes, indeed, given 20 numbers, five were in the top quartile, five were in the second quartile, five were in the third quartile, and five were in the fourth quartile. How profound! This is sometimes published as well to “help” the consumer—get out the ouija boards!

And there was our System 19 in the absolute worst quartile of both responses (as were systems 7 and 15). We sure lost the lottery in this survey (and systems 1 and 12 were probably quick to design their new marketing materials to take advantage—maybe even a TV commercial). But maybe we'll “win” next year, or maybe another survey done by someone else will exonerate us.

Better yet, let’s send everyone to “smile... or else!” training—everyone except upper management, that is. This was a favorite solution of this marketing VP, who once said to me, “Don’t those frontline idiots know how important this is?” I was very tempted to say, “They’re only treating the customer the same way they’re being treated by you. What else could you expect?”

I left this organization shortly thereafter.

What is the cost of all this wasted energy—going to unnecessary management strategy meetings, unnecessary training, unnecessary “rally the troops” meetings—and expensive ink? As I said last time, what if it were focused on improving quality instead of getting a good score?

About The Author

Davis Balestracci

Davis Balestracci is a past chair of ASQ’s statistics division. He has synthesized W. Edwards Deming’s philosophy as Deming intended—as an approach to leadership—in the second edition of Data Sanity (Medical Group Management Association, 2015), with a foreword by Donald Berwick, M.D. Shipped free or as an ebook, Data Sanity offers a new way of thinking using a common organizational language based in process and understanding variation (data sanity), applied to everyday data and management. It also integrates Balestracci’s 20 years of studying organizational psychology into an “improvement as built in” approach as opposed to most current “quality as bolt-on” programs. Balestracci would love to wake up your conferences with his dynamic style and entertaining insights into the places where process, statistics, organizational culture, and quality meet.