Conducting Remote Assessments: Lessons Learned

With planning and flexibility, remote assessments can be as effective as those done in person

Published: Tuesday, December 8, 2020 - 13:03

Recently there have been a number of articles whose authors discuss conducting assessments remotely because of the pandemic. This article will discuss my experience with an actual assessment conducted remotely, how the plan changed in response to the pandemic, lessons learned, and some of my perspectives on the assessment process. Although the lessons learned are based on an internal self-assessment, they may be applicable to other types of assessments, e.g., external, suppliers.

I led an internal self-assessment that had to be done remotely because the team and staff to be interviewed were not allowed into the workplace due to the Covid-19 pandemic. A draft of the assessment plan had been prepared before the pandemic occurred. The assessment was to be conducted during the April-May 2020 time frame.

When the workplace was closed in March (except for a small number staff deemed “essential”), a decision had to be made about how to proceed. As the lead assessor, I thought trying to perform this remotely would not be efficient or effective. I also thought the pandemic would end in a few months. However, the manager of the owning organization of this self-assessment didn’t want to wait a few months and wanted the assessment to proceed via remote means. The team and all the staff interviewed were all working from home, and all had limited access to their onsite work area. Thus, the assessment was conducted using one of the many available platforms to conduct video conferencing via the internet. Any platform used must have, as a minimum, the ability to share documents and computer screens and at least the ability to record the session, even if you choose not to.

I break down the assessment process into three phases—planning, conducting, and closing out—and illuminate changes made and lessons learned for each phase.

Planning

A written and approved assessment plan should be prepared. For any assessments, it is critical to clearly articulate the purpose and scope of the assessment. Because there was limited access to work areas, it was not possible to witness the operation of certain systems being assessed; thus, purpose and scope had to be adjusted to reflect this restriction.

A note should be added explaining that the assessment is being conducted remotely because of lack of access to the workplace due to the Covid-19 pandemic.

It is customary to have an opening meeting to go over the purpose and scope of the assessment. In my opinion this is similar to a bon voyage party: What’s the point when nothing has happened yet? Once the ship returns, then there’s something to talk about. So the opening meeting was deemed unnecessary, but a closing meeting was definitely needed. Instead of an opening meeting, a copy of the approved assessment plan (with lines of inquiry) was sent to all staff to be interviewed as well as their management. Because this was an internal assessment, the team was familiar with the staff of the organizations being assessed. This also lessened the need for an opening meeting. If you are dealing with a supplier or external assessment rather than an internal assessment, the team will need to meet with counterparts, so in those cases an opening meeting is advisable.

A team should not be concerned that sending out the assessment plan in advance will allow the organization to fix problems before the assessment starts. A competent team should be able to spot “fresh paint” to address issues, and this would be noted in the final report.

Since staff were working remotely, the ability to retrieve documents, (e.g., operating procedures, records) was an issue. The plan should stipulate a date by which documents should be retrieved. Also, it should state that if the documents aren’t produced by the due date, then a nonconformance will be issued. In that case, the nonconformance should be worded to state that requested documents “were not produced,” rather than state they “don’t exist,” since we don’t know that yet.

In our case, the difficulty in retrieving documents wasn’t initially considered, and a due date was not included in the plan, which did cause problems. Therefore, a deadline was established and communicated to staff during the conducting phase of the assessment.

Normally, an assessment is scheduled over a time frame of one to five days. This is usually necessary for assessors who are located out of town. However, with the ability to schedule interviews remotely, they didn’t have to be scheduled over several consecutive days. In this case, the interviews were spread out over two weeks and typically no more than two in one day. This was beneficial to the assessors because it enabled them to more easily keep up with their regular work.

Also, when an assessment is scheduled for several consecutive days, the assessors may be stressed in trying to maintain the schedule. Invariably, something will happen, e.g., needing more time for an interview. Spreading out the interviews reduces stress on the assessors and may result in a better assessment when time is less of a factor. Also, not being able to maintain a schedule may be perceived as incompetence, and that will turn into a lack of credibility for the team. Once a team loses credibility, their report will not be taken seriously.

It is customary to have a debrief meeting at the end of each day for the team to discuss their findings with management. In light of the limited interviews each day, the need for a daily debrief wasn’t warranted, and it was decided that there wouldn’t be any debriefs while the assessment was being conducted. Because there was no debrief, where you would normally address nonconformances, any nonconformances were corroborated while the assessment was still in progress, per proper assessment protocol. This is necessary so there are no surprises at the closeout.

Conducting

From my perspective, there didn’t seem to be much difference between conducting remote interviews vs. in-person interviews. During this phase the platform we used allowed each party to see and hear each other. It also allowed for sharing screens to display things like procedures. In fact, it seemed these interviews were more relaxed and less tense than in-person interviews; perhaps because everyone was in the comfort of their home.

As expected, when staff were asked for various documents (including records), they could not be produced on the spot. The time to receive these documents ranged from one to 10 days. As mentioned in the planning section, a due date had to be established and communicated, in the middle of this phase, for staff to produce the documents. To keep everyone at ease, these interview sessions weren’t recorded.

Toward the end of this phase, I started experiencing some audio and video problems with the platform. Most of the current platforms will run on a smart phone, which was so in my case, so that served as a back-up. This should not be troublesome in the future because, unlike most hardware, software actually gets better with time.

Closing out

A closeout meeting was scheduled. Because this was being done remotely, the scheduling of the closeout was more flexible than an onsite one. Unfortunately, during closeout we used a different platform that did not allow us to see each other, but we could still interact verbally. Delivering a closeout without being able to see attendees’ faces and body language was disconcerting because it wasn’t possible to gauge how the closeout was being received. During a live presentation, the presenter needs to talk to and see the audience to keep them engaged. In this situation, there was no choice but to talk to the screen. If you must do a closeout remotely, it is advisable to use a platform where the participants can be seen as well as heard.

Conclusions

Based on this remotely conducted assessment, the following are the lessons learned:

• During the planning, be conscious of limitations that might impact the objectives and scope of the assessment because it is being done remotely. As noted in this real assessment, some systems could only be seen in person, so these couldn’t be included in the scope.
• A due date for documents and records to be produced needs to be part of the plan.
• The ability to spread out the assessment over a few weeks may be an advantage to the team and the assessed organization.
• Being able to draft the assessment report can be easier when done remotely as opposed to in-person meetings.
• Make clear in the plan and assessment report that because of restrictions in response to the pandemic, the assessment could only be done remotely.
• If possible, choose a platform that supports showing video of attendees when conducting opening and closing meetings. It might also be useful if it has recording capabilities.

In conclusion, even if an effective vaccine becomes available, organizations will probably perform more of their assessments remotely than before, although not everyone is enamored with them. Also, keep in mind that staff (I predict at least 25 percent) will continue to work remotely post-pandemic and thus would only be available to be interviewed remotely. Remote assessments can result in real cost savings. For example, a supplier assessment done remotely results in travel expense and time savings; an organization can either perform the same number as before and deploy their resources elsewhere, or perform more for the same costs.

In the end, I think we will find that virtual assessments can be as effective as in-person assessments and will become the preferred method. 

About The Author

Steven Stein

Steve Stein has more than 40 years as a quality professional and has worked in the medical device and R&D environments. He has been involved in analyzing business and quality systems through external/internal assessments and evaluating supply chain management programs. He was a Certified Quality Engineer (CQE) and a Certified Quality Auditor (CQA) by the American Society for Quality for more than 30 years.