Featured Product
This Week in Quality Digest Live
Management Features
Jason V. Barger
How to help your people and engage with them remotely, in hybrid and in-person settings
Dirk Dusharme @ Quality Digest
At the top of the list: Trust
Sara Adams
Here’s how you can avoid making them
Stavros Karamperidis
Ships that diverted from their usual routes during the pandemic are cutting it fine to get back to China in time to restock
Torsten Schimanski
Benefits to students, employers, and industries do more than address the skills gap

More Features

Management News
Is the future of quality management actually business management?
Sept. 16, 2021, at the Duke Energy Center in downtown Cincinnati
Morse has a long history with the center, both as a faculty researcher and for serving as the center's deputy director since 2010
Siemens introduces PCBflow, a secure, cloud-based solution for accelerating design-to-manufacturing handoff for printed circuit boards
Includes global overview and new additive manufacturing section
Tech aggravation can lead to issues with employee engagement, customer experience, and business results
Harnessing the forces that drive your organizations success
Free education source for global medical device community

More News

Steven Stein

Management

How to Perform a Supplier Evaluation

Four methods for performing a supplier evaluation, and how each method can be employed

Published: Monday, July 26, 2021 - 12:03

Supply chain management (SCM) has been defined as “the design, planning, execution, control, and monitoring of supply chain activities with the objective of creating net value, building a competitive infrastructure, leveraging worldwide logistics, synchronizing supply with demand, and measuring performance globally.”1 With the advent of the Covid-19 pandemic, the importance and, fortuitously, vulnerability of a number of supply chains have been highlighted. This is especially true with the disruption of overseas supply chains for personal protective equipment (PPE), e.g., gloves and masks, and pharmaceuticals.

Suppliers can be considered third-party organizations, but they are also extensions of your organization. One aspect of managing suppliers is to evaluate their capabilities to supply items or services that meet your requirements or specifications at a fair price, with on-time delivery, and appropriate customer service and support.

An organization must have a documented process for conducting supplier evaluations. The starting point is a risk-based process, which should be a subset of your risk management process, to determine which suppliers, pre- and post-procurement, need to be evaluated. There are risks involved in supplier conformance with organizational policies, codes of conduct, regulations, and even local laws.2 The risk-based process needs to ensure that resources are allocated properly to evaluate whether suppliers of items and services surpass a defined threshold. The process should also take into consideration the consequences of and probability that suppliers could:
• Create an environmental, safety, health, regulatory, or security (e.g., cybersecurity) issue
• Incur a monetary loss due to damage, or costs associated with repair, rework, or scrap
• Reduce the availability of a facility or equipment
• Adversely affect a program objective or degrade data quality
• Unfavorably affect your programmatic mission or the public’s perception of your mission or company

This risked-based process will conform to the Pareto Principle and allow you to determine who are the vital few suppliers (20% of your suppliers) from the trivial many (the remaining 80% of your suppliers). For example, a supplier of stationery supplies, e.g., pencils, doesn’t need to evaluated, while a supplier of an item that is part of a safety system in a nuclear reactor will.

As part of the supplier evaluation process, the frequency for evaluating a supplier must be established. Typically, the frequency is between one to three years; the frequency can be based on the risk.

Methods for performing a supplier evaluation

The following are four methods for performing a supplier evaluation, ranked from most effective/least risk to least effective/most risk.

1. Onsite or remote evaluation

This is an evaluation of the supplier’s facility, staff, equipment, and implementation of their management systems, either onsite or remotely, to confirm that the supplier meets the requirements as stated in the procurement documentation.

2. Third-party evaluation of supplier

This is an evaluation performed by another company or registration body.

3. Desktop review

The supplier’s facility and management systems can be evaluated using one of the following methods:
• Reviewing the supplier’s performance history on similar items or services previously procured
• Reviewing the supplier’s applicable documentation (e.g., quality program or manual, procedures)
• Requesting the supplier complete a supplier survey form
• Discussing with the supplier’s technical or operational personnel the requirements and criteria that must be met and recording the results (e.g., on the supplier survey form).  

Tip: A supplier survey form can be included in the request for quotation package and completed by the supplier. It could form the basis of a subsequent onsite evaluation.

4. Sole-source justification

A sole-source procurement is where there is only one supplier capable of providing an item or service, and therefore it is not possible to consider other suppliers. When a sole source is required, it can serve as an acceptable evaluation provided it addresses all the necessary evaluation criteria as stated in procurement documentation (e.g., statement of work, technical specification, and drawings).

Tip: There can be changes taking place with a supplier that may require moving up an evaluation, such as a merger and acquisition (M&A) or a change in key personnel.

These four evaluation methods are explained below in more detail.

Onsite or remote evaluation

A key decision first must be made: Will the evaluation be done onsite or remotely? Once that decision is made, any onsite or remote evaluation needs to start with a plan.

Creating an evaluation plan

The foundation of the evaluation plan is to clearly articulate what the purpose and scope of the evaluation are, and these are usually driven by what resources are available. Other components of the plan are as follows.

Schedule
Usually the schedule will start with establishing the date to send the plan to the supplier as well as the dates of the actual evaluation, including opening and closing meetings. After the evaluation is performed, a draft report is sent out to review for factual accuracy, and then a final report is issued. Consideration must be given to determine if periodic debriefs (e.g., daily) will be conducted in addition to a closing meeting. Also, if a corrective and preventive actions (CAPA) plan is needed to address issues, the time frame to submit it should be addressed in the final report.

Team members
Teams should be as small as possible. Evaluations conducted by more than five members may be seen by the supplier as a gang, not a team. However, consider including a subject matter expert (SME) is needed, for example, if special processes like welding are involved. Thus, if the item to be procured involves welding, then a SME in welding who is familiar with the mission of the American Welding Society (a source for qualification and certification of welding personnel and standards), would be needed to determine if the welding process is being properly controlled.

Evaluation criteria
This can be based on your organization’s criteria. For example, the supplier must have a quality program based on quality assurance requirements for nuclear facility applications, or the supplier’s criteria for welding must be performed per the American Welding Society standards.

Specific documents and records
Training records, incoming and in-process inspection reports, operational procedures, and processes to be observed can be listed in the evaluation plan.

Strategy
Lines of inquiry or checklists should not have yes/no questions. Instead, objective evidence must be obtained to show if a requirement is in conformance or not. Also, the definition of terms to describe if requirements are satisfied, e.g., nonconformances/findings, observations, opportunities for improvement, need to be defined. There are no standards for these terms; one team may use “finding” in a negative connotation, and another team may use it in a positive connotation. Whatever terms you use, be consistent throughout the evaluation process.

Final report format
The final report can include the following sections: executive summary; purpose, scope, and strategy; results; noteworthy practices, nonconformances and findings, observations, and opportunities for improvement; and evaluation resolution.

Pros and cons of remote evaluation

As noted above, a key decision is to determine if the evaluation will be done onsite (in person) or remotely. Conducting assessments remotely is not new and predates the Covid-19 pandemic. Remote evaluations will most likely be a fixture in an eventual post-pandemic world. Registrations to ISO 9001 have been conducted remotely for several years now. (Remote auditing is one of the audit methods described in ISO 19011:2018 Annex A1.3)

Remote evaluations can result in real cost and time savings due to no travel expenses. An organization can either perform the same number of evaluations as before and deploy their resources elsewhere, or perform more for the same costs.

However, one possible obstacle to widespread adoption of remote evaluation is the concern that issues will be missed because there aren’t “boots on the ground” to physically observe facilities and actual work. To counter this obstacle, options such as live streaming via mobile technology can be employed. It is possible to use video applications on smart devices to immerse the evaluation team. This technology enables virtual site tours. It allows the team to observe the implementation of processes and activities in real time. Remember, even being in person at the supplier doesn’t ensure that nothing will be missed. Just as 100-percent inspection is not 100-percent effective, there is always a risk of not seeing everything, regardless of how the evaluation is conducted.

Finally, if you know what you are looking for, going in person or virtually shouldn’t matter. Conversely, if you don’t know what you are looking for, it doesn’t matter if it’s in person or virtually. Also consider that many suppliers have conditions that are adverse to safety (e.g., active forklifts, slipping and tripping situations), health (noise, chemical exposure), and comfort (hot or cold temperatures, access to toilet facilities). With a virtual evaluation, most adverse conditions to safety, health, or comfort can be avoided.

Ultimately, virtual assessments can be as effective as in-person assessments and will mostly likely become the preferred method.

Tip: The evaluation can be a hybrid. conducted both onsite and remotely. For example, a subset of the team can perform part of the evaluation onsite for a just a day or two, and then the rest of the evaluation could be done remotely.

Evaluating a quality program

The primary reason for evaluating a quality program is to make sure the program is accomplishing its intended functions effectively and economically. The following expounds on how to evaluate a supplier’s quality program.

Quality can be expressed in terms of technical excellence, management capability, personnel qualifications, prior experience, past performance, and conformance of supplied items or services. Examples of areas to focus on include the following:

Program
Program evaluation judges the effectiveness of a program on the basis of conditions under which it operates. It considers such things as space, facilities, and the number and type of employees. This evaluation determines if facilities, personnel, and the program itself are operating effectively.

Item or service
This evaluation consists of gathering evidence that indicates the degree of conformance of the item or service to the stated design, and evidence that the design truly satisfies the design intent. This type of evaluation is useful for mass production involving a volume of identical items over a long period, e.g., commercial off-the-shelf (COTS) items.

Process
Process evaluation emphasizes things that the supplier, the workers, and the machines do in producing an end item or service. It assumes that certain methods of operation will lead to desirable results, and that such results can be most effectively obtained when activities are directly related to goals. This evaluation requires examining the overall program for evidence relating to (a) setting of process performance goals; (b) existence of adequate procedures for obtaining goals; (c) determining the extent to which the goals are achieved; and (d) improving the process when a need is indicated.

Tips
Here are several tips to make the quality program evaluation effective and efficient.

When reviewing records and documents, instead of completing the objective evidence section in a checklist (e.g., purchase orders, inspection reports, procedures), take photos with your smartphone, with the permission of the supplier. Then, at later time, you can complete the checklist by viewing the photos.

Use the technique of process tracing. For instance, an environmental laboratory is being evaluated. The lab will need to procure standards to calibrate an instrument. Request a recent analysis report and see what standards were used (usually there will be a lot or serial number). Then request the purchase order, incoming inspection report, record of preparation of standard (per a procedure), and disposition record concerning this standard. This tracing looks at the supplier’s procurement, incoming, and in-process record, and operational procedures processes.

To back up your checklist and draft evaluation report, consider emailing them to yourself each day.

Remember, there should be no surprises. Always corroborate any evidence that a requirement is not being met. Presenting false issues will be detrimental to the credibility of the evaluation.

Supplier housekeeping can be an important aspect of the evaluation. If there is clutter, items in disarray, or lack of cleanliness, this may be the harbinger of bad tidings. This can be a strong indication that things are or will be out of control.

From a holistic point of view, in addition to evaluating a supplier’s quality program, it is recommended to ascertain how the supplier protects their workers and the environment. For example, does the supplier have in place programs similar to ISO 45001, which is an occupational health and safety management system, or ISO 14001, an environmental management system? These optional topics can be evaluated to the depth that you choose, e.g., just a cursory look to determine if there are any formal systems and no determination of implementation. Also, the financial health of the supplier can be evaluated.

Evaluating a service supplier

Evaluating a supplier that provides a service isn’t as straightforward as a supplier of any item because a service is usually not a physical thing. Two common procured services are architectural and engineering (A&E), and calibration services. Specific evaluation criteria should be tailored for each service procurement. Examples of criteria include:
• Experience and technical capability
• Facility and equipment capability
• Past performance (history)
• Environmental, safety, and health competence and performance
• Reputation and standing
• Workload capacity and current workload
• Cost

Tip: For a calibration service supplier, the supplier should hold accreditation to ISO/IEC 17025—“General requirements for the competence of testing and calibration laboratories.”

Third-party evaluation of a supplier

A third-party evaluation of a supplier is performed by an independent, external organization (a registrar, for instance) against a recognized standard. For example, if ISO 9001 is used to justify a supplier, the registration needs to be verified that it is currently in effect. Also, the ISO 9001 registrar should be under the auspice of an accreditation body.

Why it’s important to verify certification with the registrar and confirm that the registrar is under the auspice of an accreditation body is explained in the following two examples.

An organization had qualified a supplier based on the supplier’s registration to ISO 9001:2008. When it was time to requalify the supplier, the supplier sent an updated certificate showing they had been recertified. This cycle was repeated several times. In September 2015, the standard was updated to ISO 9001:2015, and there was a three-year transition period ending in September 2018, after which all organizations could only be registered to ISO 9001:2015. The supplier needed to be requalified in 2019. The supplier sent a certificate showing they had been recertified to ISO 9001:2008. As noted above, this was not possible. The registrar was contacted, and they stated they had ceased being the supplier’s registrar just before the supplier’s first recertification visit was to take place. The supplier had clearly forged all the certificates since the very first one they had supplied.

In another example, a supplier provided a certificate of registration to ISO 9001. A review of the certificate revealed that neither the name of the registrar nor the person who signed it was clear. Also, the certificate didn’t show that the registrar was under the auspice of an accreditation body. During a phone call to the supplier, they stated that the registrar had, in fact, just completed the supplier’s requalification assessment and were about to issue a revised certificate on the spot. It turns out that this registrar, which consisted of a single person, had developed the supplier’s quality management system and then turned around and assessed and then approved the supplier, thus blurring the line between being a consultant and a registrar. This is basically a violation of a “prime directive”: You can’t fulfill both roles and remain independent and unbiased. One of the reasons the registrar must be under the auspice of an accreditation body is to prevent the same registrar staff performing both the supplier assessment and issuing the certificate. The assessment team would have to submit a report to independent staff of the registrar to determine if the supplier has met the criteria to be certified.

A side story: A supplier was certified to ISO 9001. The qualifications for the quality manager were requested. The qualifications consisted of a five-day course on being a lead assessor to ISO 9001. A quality manager needs more than this to be qualified.

Certifying a supplier based on a certificate to some standard is not a “check the box” exercise; a thorough review of the certificate is required. Most registrations are legitimate; the two examples above are rare. The point again is sometimes you may want to see what’s “behind the curtain.” The moral, you must do diligence on any certification of registration to any standard; trust but verify.

Tips
• It doesn’t make sense to pre- or requalify a supplier unless there is an imminent need to procure a specific item or service.
• Ensure that the scope of the registration matches the item or service you’re looking to procure.
• Confirm with the accreditation body that the registrar is qualified to certify to the standard and scope listed on the certificate.
• The International Accreditation Forum (IAF) has a new international database (IAF CertSearch) for QMS certifications to assist in determining if a registration certificate is valid.

Certain groups or industries share their evaluations with each other:

Nuclear Procurement Issues Corp. (NUPIC)4 shares evaluations among its membership. NUPIC has established a Joint Audit Program based on an industrywide standardized approach to performing supplier evaluations. The evaluations are conducted with an approved, standardized checklist using performance-based methodology consistent with EPRI NP 6630—“Guidelines for performance-based supplier audits.”

Supply Chain QA Task Group (SCQTG)5 serves members of the Energy Facility Contractors Group (EFCOG) by identifying, analyzing, assessing, and recommending methods that support effective implementation of supply chain quality (SCQ) requirements. The group also resolves SCQ program performance issues. In particular, the SCQTG focuses on improving assessments of suppliers across Department of Energy (DOE) sites. Accordingly, the group engages in regularly planned meetings to foster consistency of supplier evaluations across DOE sites, enabling cost avoidances by sharing supplier evaluations.

Check to see if there is already a program similar to NUPIC or SCQTG, or start your own.

Desktop review

A desktop review is a high-level documentary review of policies and procedures. It is designed to verify that an organization has developed and (at least on its face) implemented effective policies, procedures, and practices.

A desktop review examines the documents within the review scope. It is referred to as a “desktop” review because it can be performed remotely at the reviewer’s desk. The documents are reviewed to ensure they are complete, correct, consistent, and current. The information gained may also be used to plan for the on-site evaluation, if that is deemed necessary. The desktop review also gives an indication of the effectiveness of the supplier’s document control system.

A desktop review can be done in several ways:
• Phone survey (completed by your organization)
• Email or regular mail survey (completed by the supplier)
• On-site survey (completed by your organization)

One vehicle that can be used to perform this review is a Supplier Survey Form. It can consist of the followings headings and some typical components for each heading:


Headings

Components

Company profile

Address; Phone numbers, Socioeconomic class; Products/services; Facility description; Accreditations or certifications, e.g., ISO 9001, ISO14001, ISO/IEC 17025

Key personnel

CEO/Owner; Manager of QA; Manager of manufacturing; Manager of purchasing

Customer references

Name of contact; Title; Company; Phone number

Documentation request

Quality Manual; Organization chart; Plant layout; Procedures & Work Instructions; Facilities and Equipment list; Records

Supplier systems and processes

Quality management system; Environmental management system; Occupational health and safety (OHS) program

Sole-source justification

A sole-source procurement means that only one supplier (or source), to the best of the requestor’s knowledge and belief, is capable of delivering the required item or service. Similar types of items and services may exist, but only one supplier is acceptable to meet a specific need. This should be a rare occurrence.

Possible justifications for a sole-source purchase can be based on the following situations:
• The requested item or service is an integral repair part or accessory compatible with existing equipment.
• The requested product is essential in maintaining operational continuity or to remain in conformance with established organizational standards.
• Only a particular brand or “make” is compatible with existing equipment or inventory.
• Only one supplier is known for a specialized item.
• Only one supplier is capable of providing the required service.

A sole-source justification documentation should include:
• A clear statement of the unique performance factors of the product or supplier specified, e.g., unique technical expertise or operational capability
• Why those unique factors are required
• What other products or suppliers were evaluated
• The reasons for rejecting the other products or suppliers (price should not be a justification for a sole-source award)

For an item or service that is commercially available—off-the-shelf or built to manufacturer’s specifications—the supplier may not allow any kind of evaluation to be performed. This will be noted in the justification documentation, and also incoming inspection or testing-acceptance criteria will be included.

Note: Some catalog items are “made-to-order” and are not considered to be COTS for the purpose of deferring an evaluation.

Tip: It is possible to combine several methods to evaluate a supplier at the same time. For example, the supplier could be requested to complete a supplier survey form and supply a third-party certification.

Conclusion

An essential part of your supply chain management is evaluating your suppliers. This is predicated on determining which of your suppliers need to evaluated, which is established using a risk-based process. This article has presented four methods for performing a supplier evaluation and some detail on how each method can be employed. It is important to remember that evaluating a supplier is actually a preventive action to ensure that the supplier can meet your requirements or specifications at a fair price, with on-time delivery, and with appropriate customer support. Given the increased use of remote evaluations, supplier evaluations are going to be more effective in terms of reducing costs, resources, time, and safety concerns.

The intent here is to cover key elements that are the foundation for each of the four method. Numerous resources are available for more in-depth study, including Lance B. Coleman’s The ASQ Certified Quality Auditor Handbook, Fifth Edition (ASQ Press, 2020), and Dennis R. Arter’s Quality Audits for Improved Performance, Third Edition (ASQ, 2002).

References
1. Cornell Engineering. Supply Chain: School of Operations Research and Information Engineering. Cornell University website.
2. Pearce, Gerard, and Jamison, Phil. “Taking Control of Supplier Quality.” Quality Digest, Dec. 7, 2012.
3. ISO 9001 Auditing Practices Group. “Guidance on Remote Audits.” ISO and IAF, April 16, 2020.
4. Nuclear Procurement Issues Corp. NUPIC website.
5. Supply Chain QA Task Group. Supply Chain Quality Task Team. Energy Facility Contractors Group (EFCOG) website.
6. Supply Chain Management. Sole Source Justifications. University of California, Davis website.

Discuss

About The Author

Steven Stein’s picture

Steven Stein

Steve Stein has more than 40 years as a quality professional and has worked in the medical device and R&D environments. He has been involved in analyzing business and quality systems through external/internal assessments and evaluating supply chain management programs. He was a Certified Quality Engineer (CQE) and a Certified Quality Auditor (CQA) by the American Society for Quality for more than 30 years.

Comments

2nd or 3rd Party

Hello Steven:

Why would a supplier be considered a third party? Who are the 3 parties in this structure?

First party: An organization. No other parties are involved.

Second Party: An organization and another organization, often a supplier.

Third Party: The organization, a supplier, and the benefactor of the outcome of the audit or project, often a customer or multiple customers. 

Thank you, Dirk van Putten

Meaning of a Third-Party Supplier

Dirk,

I understand your dilemma; by talking about how suppliers can be considered third-party organizations, it implies there is first and second party and they are not defined.

I believe the term third-party supplier has been around for some time; an article by Deloitte, titled ‘Third party suppliers background due diligence’, uses the term: “third party suppliers). (https://www2.deloitte.com/cn/en/pages/finance/articles/deloitte-bis-2-po...)

The following is a definition:

A 'third party', as defined in OCC 2013–29, is any entity that a company does business with. This may include suppliers, vendors, contract manufacturers, business partners and affiliates, brokers, distributors, resellers, and agents. (https://en.wikipedia.org/wiki/Third-party_management)

 

I would explain the three parties as:

First-Party = Your Organization

Second-Party = Your customer (mostly external, but could be internal)

Third-Party = Your supplier

 

The important take-away is that a Third-Party supplier can impact your customer (Second-Party) in a positive or negative way, and they are independent from your organization (First-Party), and are usually not known to your customers.