ISO 9000 Registrar Customer Satisfaction Survey


How Do
Registrars
Measure Up?

by Dirk Dusharme, Chris Costello and Thomas Pyzdek

This first comprehensive look at customer satisfaction with ISO 9000 registrars contains some surprising results.

Despite all the grumbling about the ISO 9000 registration process that has always followed the registration industry, clients are actually satisfied with the level of customer service provided by their registrars, according to Quality Digest's recent survey of ISO 9000-registered sites in the United States and Canada.

Registrars Surveyed

The following registrars' clients were contacted to take part in the survey. Registrars listed in bold type appear on one or more charts in the article.

  • ABS Quality Evaluations
  • AFAQ-Ascert
  • AGA Quality
  • AIB Registration Services
  • American Association for Laboratory Accreditation
  • American Quality Assessors
  • American Systems Registrar
  • AOQC Moody International
  • AQSR International
  • ASME International
  • BNQ System Registration
  • British Standards Institution (BSI)
  • Bureau Veritas Quality International (BVQI)
  • Canadian General Standards Board
  • CRS Registrars
  • Deloitte & Touche LLP Quality Registrar
  • DNV Certification
  • DQS
  • Eagle Registrations
  • Entela QSRD
  • EQR
  • Factory Mutual Research
  • HSB Registration Services
  • International Certifications
  • International Quality Systems Registrars
  • ITS Intertek Canada
  • ITS Intertek Services
  • KEMA Registered Quality
  • KPMG Quality Registrar
  • Lloyd's Register Quality Assurance
  • National Quality Assurance (NQA USA)
  • National Standards Authority of Ireland (NSAI)
  • National Technical Systems
  • NSF International Strategic Registrations
  • Orion Registrar
  • OTS Quality Registrars
  • Performance Review Institute
  • Perry Johnson Registrars
  • Professional Registrar Organization
  • Quality Certification Bureau (QCB)
  • Quality Management Institute (QMI)
  • Quality Systems Registrars (QSR)
  • QUASAR
  • Scott Quality Systems Registrars
  • SGS International Certification Services
  • Smithers Quality Assessments
  • SRI Quality System Registrar
    • TRA Certification
  • TÜV Essen
  • TÜV Management Service
  • TUV Rheinland of North America
  • Underwriters Laboratories
  • Underwriters Laboratories of Canada
  • United Registrar of Systems (URS)
  • Vehicle Certification Agency (VCA)

 

In this first-of-its-kind survey conducted in conjunction with Pyzdek Management Inc., SPSS Inc. and Scantron Technologies, Quality Digest provides the first comprehensive look at how the registration industry is viewed by its clients. This survey is intended to show the correlation of the various aspects of registrar/client interaction to the perceived level of customer service.

As a magazine that has covered ISO 9000 from the beginning, we have heard all the complaints and rumors about particular registration practices or registrars. We expected to discover a wide range of customer satisfaction levels--from highly dissatisfied to highly satisfied.

To our surprise, we discovered that the level of customer satisfaction is quite high. Registrars scored well above average in every category, and when clients were asked if they would recommend their registrar to others or choose that registrar again, the average survey response was a very respectable 4.19 on a five-point scale. Even those registrars that fell below the 4.19 mean performed well, with scores that ranged from 3.5 to 3.97.

To conduct the survey, Quality Digest first created a list of 53 items compiled from pre-survey interviews with more than 30 randomly selected clients. All items, except for two about overall satisfaction, came from the clients themselves.

Quality Digest then faxed questionnaires to more than 15,000 registered companies in the United States and Canada. We received more than 1,800 responses, a 12-percent response rate. All participants were asked to respond to the 53 statements related to their registration or auditor experiences. Respondents evaluated each statement on a five-point scale from "strongly disagree" to "strongly agree," with a sixth response for "doesn't apply." For a complete look at the survey methodology, see "Survey Methodology."

About the data

The following series of charts illustrates registrar performance in several customer satisfaction categories. A mean and a list of companies that performed significantly above the mean is given for each category. There is 95-percent certainty that the actual average isn't off what's shown on the charts by more than 0.05. For about half of the categories, the interval is closer to 0.02 in either direction.

Our first impulse was to simply present all the registrars by name with the data collected for each. This seemed an easy choice because all registrars performed well; i.e., the mean was high and no registrar scored poorly. However, when we discussed the results with the registrars prior to publication, those that didn't perform as well as others were mortified, arguing that in a business as highly competitive as theirs, the slightest perception that they ranked below an industry mean would "kill" them. The head of one large registrar told us that his company has been working hard during the last six months to improve customer service and he feared that "heads would roll" at his office if there was any perception by upper management that the business wasn't meeting its goal of outstanding customer service.

After much discussion, we agreed that some readers might perceive registrars that fell below the mean as "bad" companies--a perception that, although far from the truth, could indeed damage those companies.

The compromise is to present the above-the-mean performers as benchmarks for the others. The data for those registrars below the mean is also shown, but without the company names. We believe that this data is necessary to provide a true sense of the point spread in customer satisfaction, which in most cases was small.

In addition to the eight to 10 registrars depicted above and below the mean for each category, about 25 registrars fall into the mean. For a complete list of registrars for whom survey results were collected, see "Registrars Surveyed" sidebar.

Correlating the data

One of the first things we hoped to discover was the correlation between individual survey categories and overall customer satisfaction. We define overall customer satisfaction as the willingness of clients to recommend or reselect their registrar. Survey items were divided into the following categories: administration, including responsiveness, scheduling and price; communication; helpfulness, including ease of contact and useful suggestions; industry knowledge; professionalism; and thoroughness.

reg1Surprisingly, the highest correlation to customer satisfaction was in the administrative, communication and professionalism categories (see Table 1). The nuts and bolts of registration--industry knowledge and thoroughness--had less of a correlation. What this could mean is that ISO 9000 is still considered more a necessity than a choice. An "If I have to get this done, at least make it pleasant" attitude may predominate.

Looking at correlations by the 53 items rather by than categories, we see that fairness, responsiveness and professionalism are the best indicators of customer satisfaction (see Table 2).

reg2

Despite the complaints we commonly hear about how expensive registration can be, price had little impact on a client's decision to recommend or reselect his or her registrar (a correlation score of 0.196).

We were initially concerned that the size of the client company or the size of the registrar might have some bearing on customer satisfaction. In fact, this wasn't a factor. Satisfaction levels seem to be consistent regardless of registrar size or client company's size.

The survey indicates that responsiveness, a firm handshake and patience with a smile may mean more to customer satisfaction than a thorough auditor who knows the client's industry inside out.

None of the categories nor items showed exceptional correlation scores. This could mean that the ultimate customer satisfaction questions weren't asked, but since the survey items were generated by the clients themselves, it is more likely that customer satisfaction in the registration field is too nebulous to be categorized.

Administration   (Fig.1)

admin
The registrars that ranked above the mean in administration are TÜV Essen of San Jose, California; Orion Registrar of Arvada, Colorado; Quality Certification Bureau of Edmonton, Alberta, Canada; Quality Systems Registrar of Herndon, Virginia; SGS-ICS of Rutherford, New Jersey; DNV Certification of Houston; and ITS Intertek Services of Boxborough, Massachusetts. (See Figure 1.)

This was the highest-ranking category in terms of correlation with overall satisfaction. Within this category, the highest correlation item was responsiveness, followed by the timely return of phone calls and follow-up administration.

That administration had the highest impact isn't completely surprising. Administrative personnel are the first and last line of contact with the client. Pricing and scheduling start here, and billing and issuing a certificate end here. This survey indicates that a good administrative interface goes a long way toward heightening the perception of good customer service.

Communication  (Fig. 2)

comm
The registrars that ranked above the mean in communication are Orion Registrar Inc. of Arvada, Colorado; TÜV Essen of San Jose, California; DNV Certification of Houston; SGS-ICS of Rutherford, New Jersey; and ITS Intertek Services of Boxborough, Massachusetts. (See Figure 2.)

This category includes items such as sharing information, communicating with the client and with auditors, and ease of contact. With a mean of 3.99, this was the highest-scoring category in the survey. The high scores, coupled with the second-highest correlation to repurchase or recommend, make this category very important.

Because registration is such an expensive and complex process, subject to the scrutiny of managers and stockholders alike, communication is a priority. It's no coincidence that four of the five companies that scored above the mean in communication were also four of the five companies that scored highest in overall customer satisfaction: Good communication goes a long way toward relieving stress.

Professionalism       (Fig. 3)

prof
The registrars that ranked above the mean in professionalism are Orion Registrar of Arvada, Colorado; SGS-ICS of Rutherford, New Jersey; and DNV Certification of Houston. (See Figure 3.)

Overall, registrars did extremely well in this category. For nearly two-thirds of the items in this category, registrars were rated higher than 4. Important baseline characteristics like the auditors being "professional," having "good cross-functional knowledge" or being "technically qualified" ranked highly, as did less specific attributes like patience and fairness on the parts of the auditor and the audit, respectively. This seems to suggest that the caliber of auditors hired by registrars is quite high.

As with "industry knowledge," "professionalism" had a narrow range of customer satisfaction scores (0.36). This, too, seems to indicate that registrars are hiring well.

Thoroughness          (Fig. 4)

thourough
The registrars that ranked above the mean in thoroughness are Orion Registrar of Arvada, Colorado; SGS-ICS of Rutherford, New Jersey; and DNV Certification of Houston. (See Figure 4.)

This category covered such items as the objectivity and thoroughness of the audit. The fact that this category had a high mean score indicates that auditors are doing what they are supposed to do: going through the clients' processes with a fine-toothed comb.

Industry knowledge  (Fig. 5)

industry
The registrars that ranked above the mean in industry knowledge are SGS-ICS of Rutherford, New Jersey; National Quality Assurance USA of Acton, Massachusetts; and DNV Certification of Houston. (See Figure 5.)

The correlation of this category to customer satisfaction was low. However, the item regarding the application of ISO standards to a specific business stood out in terms of its rating with clients (4.08) as well as with its correlation to reselect/recommend (0.523).

Although seemingly unimportant in terms of customer satisfaction, these results do shoot down a complaint we frequently hear: Auditors don't understand how ISO 9000 applies to a particular industry.

Industry knowledge was one of the lowest-scoring categories (3.63) but also had the narrowest range of customer satisfaction scores (0.37). This may point to a consistency throughout the registration industry in hiring auditors who are more or less equally knowledgeable in their particular industry sector.

That said, the low scores should tell registrars that this area needs work.

Helpfulness    (Fig. 6)

help
The registrars that ranked above the mean in helpfulness are BNQ System Registration of Montreal, Quebec, Canada; Orion Registrar of Arvada, Colorado; ITS Intertek Canada of Lachine, Quebec; DNV Certification of Houston; and QMI of Mississauga, Ontario, Canada. (See Figure 6.)

This is a contentious, politically sticky category: The helpfulness category contained several items regarding coaching or making suggestions. According to ISO/IEC Guide 62, although auditors or registrars can provide general information that could be helpful, they cannot make specific suggestions to clients--how to correct a nonconformance, for instance.

Because we don't know what a client may have meant when they responded to an item such as "The auditor gave suggestions on how to strengthen weak areas," we don't know whether specific auditors are violating Guide 62's prohibition on providing consulting, or simply providing helpful information as allowed by the guide. Scores below the mean could indicate that some registrars err on the side of caution, being careful to present just the facts.

The wide range of registrar scores in this area raises an important question: Are certain registrars perceived as unhelpful because of their strict interpretation of Guide 62? If so, then are those with higher scores using a more lax interpretation?

Obviously, clarification of Guide 62 is needed, a fact that Joseph Dunbeck, CEO of the Registrar Accreditation Board, acknowledges. "Guide 62 is, unfortunately, not very explicit on what it means by consulting," says Dunbeck. "As accreditors, we are trying to get CASCO [ISO/IEC Committee on Conformity Assessment] to clarify Guide 62."

The president of one registrar told Quality Digest that he believes clients want and deserve direction for improving their processes and that this can be achieved without violating Guide 62. This registrar plans to "push Guide 62 to the limit."

Rather than worry about the specifics, it's more important to note that registrars ranked reasonably high in helpfulness and leave it at that. Without interviewing the clients, it's impossible to speculate further.

Despite the debate over Guide 62, helpfulness as a category had little impact on satisfaction.

Overall customer satisfaction           (Fig 7)

overall
Regardless of a client's feelings about particular aspects of an audit, the real test is whether clients would recommend their registrar to another person and whether they would choose that registrar again. The mean response to those two survey items was 4.19--on average, clients indicated their willingness to recommend or reselect as between "agree" and "strongly agree."

Figure 7 shows the overall response to the following two items:

"If I had to do it again, I would recommend this registrar to others."

"If I had to do it again, I would choose this registrar."

The registrars that ranked above the mean for overall customer satisfaction are Orion Registrar Inc. of Arvada, Colorado; TÜV Essen of San Jose, California; NSF-ISR of Ann Arbor, Michigan; SGS-ICS of Rutherford, New Jersey; and DNV Certification Inc. of Houston. (See Figure 7.)

It's interesting to note that these registrars span the entire range of sizes as judged by number of clients. Orion is a relatively small registrar with about 100 registered sites, whereas DNV has more than 3,200 registered sites.

Notice that the registrar farthest below the mean scored a 3.54 (where a 3.0 equal neutral) on these two items. This means that its clients still have a positive opinion on recommending or reselecting this registrar.

On the trail of continuous improvement

This survey indicates that the state of the registrar industry with respect to customer satisfaction is good overall, with several outstanding performers in each category.

The weakest category is helpfulness, although, as explained above, this category is problematic because it involves an interpretation of Guide 62.

As indicated by the industry knowledge and professionalism categories, hiring practices across registrars are consistent, with each registrar receiving close to the same amount of customer satisfaction for these two metrics.

The intent of this survey was to not only provide readers with some idea of how the industry was doing and who some of the key performers were, but also provide the industry as a whole with some idea of how each registrar stacks up to the next. To encourage the registrars to use this survey as we intended--that is, as a tool for continuous improvement--we provided each registrar with the complete survey results and a listing of its rank within the industry.

With one exception, all registrars we spoke to about the survey, including many who performed below the mean in one or more categories, were excited about the results and believed that the survey will be a useful tool. As one put it: "We do customer satisfaction surveys all the time, and the results are good. But whenever we show them to management, they always ask us, 'Yeah, but how is our competition doing?' Now we know."

This survey scared a few registrars. But in the end, registrars shouldn't be afraid to have the same level of scrutiny applied to them as is applied to their clients. Terms like "best practices" and "benchmarking" practically got their roots in the quality industry. This survey is intended to be a yardstick by which registrars can objectively assess their service, not one to be rapped across their knuckles. When used in that context, this information can be a good tool to improve registrar quality.

This data is a start. Let's use it well.

 

Survey Methodology

The survey methodology was designed by Thomas Pyzdek of Pyzdek Management Inc. The survey design incorporated recognized techniques for minimizing bias and optimizing both validity and reliability, throughout.

All survey items were derived from client interviews except for two on recommending or reselecting a registrar. Quality Digest interviewed 30 randomly selected registrar clients and posed the following two questions: What were the most important positive features or characteristics of your registrar? What were the most important negative features or characteristics of your registrar? The order of the questions was switched with each interview. In all, clients generated about 125 characteristics.

Quality Digest then developed categories and grouped each characteristic in an appropriate category. The categories were tested for reliability. The number of characteristics was then narrowed to 53 by throwing out repetitive items or those that didn't fall easily into a category.

The survey form was designed using e-listen software from Scantron Technologies, which allowed us to generate a Web-based form and paper survey simultaneously. To detect misleading surveys, those in which respondents randomly marked the items or simply put check marks down one row of responses--we asked the same question twice in several instances, one phrased negatively and the other positively.

We then faxed a one-page survey invitation to every registered company in the United States and Canada for which we had a fax number--about 18,000 sites. Of those, 15,000 clients actually received the faxes.

Each respondent was given a unique identifier generated by e-listen and asked to either go to a Web site to take the survey or request that a paper survey be faxed to them. The Web site was hosted by Scantron Technologies using e-listen's autohosting feature. About 1,450 respondents took the survey online, and 350 respondents participated via fax.

The unique identifier kept unqualified personnel from participating and prevented ballot stuffing, as only one survey per identifier was allowed. Once the data was collected, the unique identifier allowed us to identify the respondent's registrar.

The statistical analysis was performed at SPSS Inc. by Chris Costello, associate manager of market research. Using SPSS 9.0 for Windows, data were subjected to reliability and analysis, analysis of variance, means and correlation testing.

 

[QD Online] [Registrars] [DOE] [Sample Plans] [ISO Software] [Federal]

Copyright 1999 QCI International. All rights reserved. Quality Digest can be reached by phone at (530) 893-4095. E-mail: Click Here