The IT department of a school district supported itself by providing application development, database maintenance, training, and other services to its internal clients, although no money exchanged hands. The school’s outreach unit, which provided training and education to local organizations, also maintained its own profit and loss statement but was having issues with invoicing. Somehow the invoiced amount always seemed to be different from the quoted amount, which meant manual corrections, delays in receivables, and most important, unhappy clients.
As one of the more experienced analysts, Joel was brought in to resolve this issue expeditiously. He quickly identified an anomaly, which led him to assume data corruption as the culprit. He dove right in analyzing data, working backward from the output data, looking for patterns. Not finding anything, he delved deeper and discovered tiny data issues surfacing randomly, and he failed to detect a pattern that would point to a root cause. None of the data defects he found affected the invoices uniformly. A supposedly quick IT project was turning out to be a nightmare, and the outreach unit was getting anxious at the number of hours it would have to pay the IT department.
The IT director brought in Angie, another senior analyst, to help Joel. Angie concentrated on the process flow of raw data for invoicing. She followed the path the data took to generate the final output of invoices, reviewing the process of collecting and converting data to check where incorrect data might have been picked up. While Joel was looking for incorrect data, Angie focused on the process—perhaps an incorrect record or double dipping—rather than assuming the data were defective. If the markup was done twice, she reasoned, the problem was a process issue.
If the analysis indicated taking a closer look at the internal rather than external customer rate, defective data wouldn’t make any difference. Detailed data analysis prior to understanding the process and how things could go wrong was simply a waste of effort. While Angie strongly believed in the value of data analysis, she also believed that a process approach would help them get to the root cause faster. And it did. Taking this approach and working together, Angie and Joel quickly discovered the process anomalies, to the delight of the customer.
Statisticians, and often Six Sigma professionals, are fond of saying, “In God we trust; everyone else bring data!” However, just delving into data might not always be the best approach. When solving problems, most people begin with an empirical approach, looking for symptoms and potential causes that can be linked to potential consequences. But whether the method focuses on data or processes, the problem must first be defined and then the Six Sigma methodology or similar process must be followed.
In the example above, the problem-solving process followed a two-pronged approach—looking at both the data and the process. The realization that there was a problem came as a result of bad output. This was caused either by bad input or good input, but either way something bad happened to it while on its way to becoming output. So where does one go from here? If taking the micro view, an analysis would begin by sorting through the data in detail. If taking the macro view, the analysis would focus on understanding the process and associated path the data take, identifying possible failure points, analyzing what could go wrong, and only then performing the data analysis.
Testing could also be done. For example, phony data could be entered into the process to see if the system generated the “right” answer. If it did, then process analysis “outside” the system would be appropriate, to see how the data are being entered incorrectly. If the right answer isn’t generated, then the error must occur internally. At this point, a detailed understanding of the process that highlights possible failure points and resulting consequences would be important.
The issue is how to get the problem-solving process defined and ingrained in the organization. Should the focus be on collecting and analyzing data first, or on understanding the workflow, identifying possible failure points, and then proceeding into data analysis? Although these are both basically problem-solving processes, unless the failure points are evident, it would appear that the latter option makes more sense.
As for the expression, “In God we trust; everybody else bring data,” without an understanding of the process and possible failure modes and points, it would be difficult to know what data to bring.
What is your approach: data first or workflow first?