Featured Product
This Week in Quality Digest Live
Management Features
Eric Whitley
Seven tips for efficient maintenance
Zhanna Lyubykh
Consequences and costs of abusive supervision
Dario Lirio
Modernization is critical to enhance patient experience and boost clinical trial productivity
Oliver Binz
Better internal information systems help managers tell consumer demand from inflationary pressure
Dale Crawford
Electrical contractors and other skilled trades are losing institutional knowledge

More Features

Management News
Attendees will learn how three top manufacturing companies use quality data to predict and prevent problems, improve efficiency, and reduce costs
More than 40% of directors surveyed cite the ability of companies to execute as one of the biggest threats to improving ESG performance
MIT Sloan study shows that target-independent compensation systems can be superior
Steps that will help you improve and enhance your employee recruitment, retention, and engagement
300 Talent acquisition leaders and HR executives from companies gather in Kansas City
FedEx demonstrates commitment to customer-focused continuous improvement
SONY-based 8MP color, UVC USB, high-speed camera provides high dynamic range and dual-stream support
Configuration lifecycle management provider saw 42% increase in annual recurring revenue
Designed to offer a comprehensive safety solution for fleet vehicles and workforce personnel

More News

Steven Severt

Management

After a Global Pandemic, Are Fully Remote QMS Audits Truly Effective?

Looking at intent, realities, and possibilities

Published: Wednesday, January 12, 2022 - 13:03

Although ISO 19011—“Guidelines for auditing management systems” has included language about remote quality management system (QMS) auditing since the 2018 revision, this became a reality for many of us in March 2020 with the onset of Covid-19 and the mass lockdowns that ensued. Many of us have been exclusively auditing quality management systems remotely for a year and a half, and without doubt will continue to do so.

In the beginning of the pandemic, there were articles circulating to point to the language of ISO 19011:2018 or blogs from consultants that claimed they were able to teach effective remote-auditing methods to ensure that the results were just as effective as their onsite counterparts. However, now that we are a year and a half into this, I wanted to get some opinions about the actual effectiveness of remote auditing as it stands today.

I wanted to know if auditors’ and auditees’ perceptions about the effectiveness of remote auditing had changed during the pandemic, or whether they felt that we were truly meeting the criteria set out in ISO 19011. I wanted to learn more about the techniques that auditors have developed to make fully remote audits more effective, and what the professional community thought we ought to do to continue to close the gap to make them more effective in the future.

The poll

So, to open the discussion, I ran a poll on LinkedIn (a very scientific study, I know) that ran for one week that simply asked the question: “After 1.5 years of global pandemic, do you find that fully remote QMS audits are truly effective?” as well as prompting the respondents to comment with their experiences, techniques, perceptions, and thoughts for the future. Respondents were given the following four options in the poll: 1) Yes, more [effective] than onsite, 2) Yes, as effective as onsite, 3) No, less [effective] than onsite, and 4) Not at all effective. While there were only 96 respondents, the results of the poll were not all that surprising.

remote audit poll

Most respondents, 52 percent, felt that remote audits were less effective than onsite audits, and after that, respondents became a bit more divided, with 18 percent taking this a step further and saying that they are not effective at all, and 26 percent believing remote audits that are currently being conducted are as effective as their onsite counterparts. Only 4 percent felt that remote audits were currently more effective.

Although the percentage of respondents saying that remote audits are currently as effective is a bit higher than expected, the data collected certainly suggest that there are some gaps to fill to make remote audits as effective as they should be, as well as effective as they will need to be as we potentially move to a more remote way of working.

What ISO 19011 says

Before digging into the whys and the solutions, I thought it prudent to take a closer look at what ISO 19011:2018 says so we can get a frame of reference from the current standard’s language on the topic.

A search of the term “remote audit” in ISO 19011:2018 will yield just six results, all of which are embedded within the body of the text and appendices. Section 5.4—“Establishing the audit program,” includes the first two.

Paragraph 5.4.3 k) basically states that individuals managing an audit program must determine the extent of the program, with factors such as the availability of information and communication technologies to support remote auditing methods potentially having an impact. Similarly, 5.4.4 g) suggests that the audit program manager must determine resources for the audit program, including the availability of information and communication technologies required to set up a remote audit and support remote collaboration.

These statements seem obvious and don’t yet provide much guidance, but respondents did sometimes echo these statements in their comments on the poll.

One respondent who had commented that he believed remote audits were “just about as effective” was asked what he thought would be needed to help close the gap so they could be as or more effective.

He states: “As strange as it sounds, the next-largest gap was connectivity throughout the plant. We had trouble running live video communication in many parts of operations. To add to the trouble, the noise made it difficult to hear, and it was impossible for the guide to hear. Some elaborate technology setup could help, but it also needs to be simple and reliable.”

This respondent’s sentiment was shared by many others and, indeed, matches closely with some of my own experiences. When planning an audit program, I think many of us are still trying to find the right balance between what needs to be conducted via livestream and what can be done remotely without the use of streaming technology. For the former, I believe that we’re all working to better understand what resources and technologies will be needed to livestream effectively, and what factors and conditions might prevent us from truly meeting audit objectives. If the ability to see and hear the environment on a shop floor is necessary for an audit or site assessment, for instance, do we truly understand how to do that effectively before we engage in the audit activity? If we haven’t worked out all of the bugs in the planning phases, will we really gather the evidence that we need to achieve the audit objectives?

remote audit quote

The aforementioned instances in which remote audit is mentioned in ISO 19011:2018 are the only times that the term is used within the body of the clauses. All the remaining occurrences appear in Annex A (informative): Additional guidance for auditors planning and conducting audits.

The third occurrence is in a note in Table A.1, which is a matrix outlining audit methods based on auditor location with these quadrants: onsite with human interaction, onsite with no human interaction, remote with human interaction, and remote with no human interaction, with the fourth in the succeeding paragraphs qualifying feasibility of remote audit activity.

For most of the respondents, it appears that most remote audits being performed are either remote with human interaction or a combination of remote with and without human interaction. The matrix gives several general methods for collecting information such as conducting interviews, observing work with a remote guide, completing checklists and questionnaires, and conducting document review with or without auditee participation, but it’s otherwise not a source of guidance on how to do so effectively.

The fourth occurrence exists in the statement, “The feasibility of remote audit activities can depend on several factors (e.g., the level of risk to achieving the audit objectives, the level of confidence between auditor and auditee’s personnel and regulatory requirements).” It goes on to say, “...it should be ensured that the use of remote... audit methods is suitable and balanced, in order to ensure satisfactory achievement of audit program objectives.” Similarly, this only restates the need to ensure that methods are used and resources determined and employed that ensure that audit objectives are met, but provides no other information on how to do so.


Image above from ISO 19011:2018 used with Fair Use Copyright for the purpose of education and commentary

Closing the gap

We had already established through the poll that most auditors using remote methods don’t feel that they are consistently meeting audit objectives through virtual means, but we have yet to gain much valuable insight on how to close the gap, either from the polling or from the text of ISO 19011.

Of those who felt that remote audits were almost as effective or as effective as onsite audits, when pressed for techniques used, they primarily focused on a combination of the proliferation of new technologies and the reinforcement of widely established auditing techniques that would otherwise be used onsite.

For the use of new technology, it seems to be widely accepted that the increased use in electronic platforms, intranet or cloud documentation solutions, and electronic quality management system (eQMS) solutions provide a huge advantage when virtually auditing.

One respondent commented: “I’ve been working in the field of quality management systems for industrial quality, as well as the digital transformation of paper QMS to software for more than 10 years, and of course this has enabled us to audit the QMS remotely, as every department has all QMS documentation electronically filled out in the system. It is found to be very effective, especially for multi-factory companies. Also, I have to mention, some parts of the electronic QMS are self-audited and followed up on by the system itself.”

It appears that this particular respondent is using primarily an approach that’s remote with no human interaction, and he didn’t respond when pressed about difficulties when human interaction was needed, or when activities at the multiple sites being audited needed to be observed. But his experience with documented elements of a QMS seems to be common, and, indeed, mirror my own.

In my experience, I’ve often used a combination of remote with and remote without human interaction to conduct QMS audits, and so long as the auditee uses and provides access to electronic systems used for document and record control, these portions of the audit can be conducted very effectively, with difficulties typically arising from technological limitations. With the widespread proliferation of eQMS, enterprise resource planning (ERP), and customer relationship management (CRM) technologies in many manufacturing, service, retail, and healthcare industries, it stands to reason that we will continue to see rapid improvement in our ability to audit documentation and records effectively and efficiently remotely.

One comment of note to help improve situations in which human interaction is necessary came in the form of a stark reminder from an experienced director of quality with more than 20 years of experience in operational excellence, and who sits as a current chair for ASQ’s Silicon Valley section:

“I just completed two supplier audits. Both were completed effectively in shorter time. With complete transparency between auditee and auditor, remote audits can be as effective. One should recognize audit is a review of a sample of the QMS, not combing through for 100-percent verification. Instead, I find auditors lacking experience in following the intent of quality auditing.”

This particular comment struck me more than any other. This professional made a statement that is completely in line with quality gurus such as Phil Crosby and W. Edwards Deming. Quality improvement doesn’t always require expensive new technology or equipment, but better utilization of the resources that already exist.

If we commit to our principles of quality improvement, we will note that many of the failings that we currently experience in remote auditing, especially when human interaction is a factor, come not just from technological limitations, but also with the limitations in our auditing abilities. We should remind ourselves that the intent of QMS auditing is merely to sample evidence to assess whether the criteria have been met, and only dig deeper when it seems that the criteria hasn’t been met.

This point isn’t lost, then, when I examine the last two instances in which “remote audit” is mentioned in ISO 19011:2018. They are both in section A.16 Auditing virtual activities and locations, and the section spends nearly equal amounts of time discussing auditor competency as it does enforcing the need to ensure that technological requirements are met.

To make remote audits truly effective, it’s clear that there are still some gaps to fill. Because it appears that remote auditing is here to stay, we owe it to ourselves to understand the gaps and start working toward solutions. I believe that remote auditing does have the potential to be as effective, if not more effective, than the exclusively onsite audits of yesteryear.

There are still some technological hurdles that need to be overcome to ensure connectivity throughout the location being audited. But perhaps more important, we need to invest in training our auditors to audit more effectively in remote scenarios.

Although most respondents unsurprisingly cast their vote stating that remote audits are not yet as effective as they should be, their shared experiences helped to illustrate a not-too-distant future in which they will be.

Discuss

About The Author

Steven Severt’s picture

Steven Severt

Steven Severt, who writes for isoTracker QMS, is a quality management professional with nearly two decades of experience in the automotive and medical device industries. He has extensive experience launching and supporting manufacturing processes to supply automotive OEMs as well as developing, supporting, and auditing quality management systems that adhere to the requirements of ISO 9001, IATF 16949, ISO 13485, and 21 CFR 820.

Comments

Article

Nice article. Good insights. Thank you. Mukund

On-site audits suck, too

I think there are a few problems with how the question is being asked. It assumes that on-site audits, as conducted over the past few decades, are some sort of baseline for effectiveness. In reality, as it pertains to ISO certification audits, all evidence shows that on-site audits are largely ineffective, and have been so for many years. 

My company (Oxebridge) has amassed a great amount of data showing that between the period of roughly 2000 to present, certification audits have fallen to a pay-to-play scheme, in which now everyone passes every audit, often with "no nonconformities." Under no logical scenario should this be possible. Even worse, when companies are later uncovered to have released defective products, or falsified inspection data, or released counterfeit products .... nothing happens. The companies continue to retain their certifications as auditors simply ignore the evidence around them, as well as the blaring news reports decrying the crimes. Again: you pay, you get the certificate.

As I write this, the new head of the IAF just wrote to me with an opinion that would allow companies found guilty of fraud to maintain their ISO certifications. And this comes from the organization with is supposed to uphold the integrity of certifications.

So the pearl-clutching over remote audits is worth a number of eye-rolls. If bodies like IAF and the attendant accreditation bodies have been unwilling to make on-site audits more than just performance art, why is anyone raising a fuss now about remote audits?

The entire assessment model needs to be fixed, starting with a return to requiring evidence by way of documents and records. This reduces auditor interpretation, and the need for elderly grunts to rummage around the shop's garbage bins in between trips to the coffee machine.

You are assuming this refers only to External audits!

While you can argue that external audits 'on-site' may not be effective; Internal Audits 'on-site' are effective.

So in one case the data is valid and in the other case, probably not so much!

Remote internal audits by

Remote internal audits by auditors who are already on site? Mmkay. 

I Was Primarily Talking About Imternal Audits

Hi, Chris.  I am the author and was primarily talking about internal audits.  I'm now doing a lot of remote audits for clients that I used to audit for onsite, or reviewing their previous internal audit reports that were conducted onsite prior to conducting my own remotely.  I definitely see your point and agree with you, but I'm really not taking about CB corruption and the IAF, just talking about challenges that regular internal auditors need to learn to overcome both by improving their skills and the tools used.

Same argument applies

It's irrelevant. The standard only demands that companies "identify" or "determine" or "plan" a thing, and then never require a record for any of those. For any auditor, whether internal or external, they would have to use super-psychic mindreading powers to obtain objective evidence of an undocumented mental exercise like "determining" something. Until the standards are updated, remote auditing can't work well for anyone. The IAF just leans into this by making audits lackadaisical drive-by events. But a lot of internal auditors do that, too. The standards don't really require anything more from them.