Featured Video
This Week in Quality Digest Live
Metrology Features
James Rawstron
Jet wing surface scan reduced to under four minutes
Michael Huda
How do you control a color that absorbs so much light, spectrophotometers can’t measure it?
Matthew Martin
Traditional CMMs are being challenged by fast-emerging, highly advanced blue-light scanning
Dirk Dusharme @ Quality Digest
Christmas is creeping, should a four-year degree really be the minimum requirement, and what we learn from Disneyland
Dean Solberg
If you can think about it and model it, you can 3D-print it

More Features

Metrology News
Four times faster data acquisition, enabling precise measurement at the submicron level
Caters to the growing need for sophisticated scanning in an entry-level solution that is budget friendly
Topics for the Go Forth and Measure project are virtually unlimited
Ask questions, exchange ideas and best practices, share product tips, discuss challenges in quality improvement initiatives

More News

Randy Long

Metrology

Validating Methods, Procedures, and Instructions for ISO/IEC 17025

Understand the standard’s intent and avoid noncompliances

Published: Monday, January 11, 2016 - 16:47

A study conducted by the Laboratory Accreditation Bureau of noncompliances during accreditation assessments to ISO/IEC 17025—“General requirements for the competence of testing and calibration laboratories” found that the most cited clauses were found in section 5.4—“Methods and method validation.” This article attempts to clarify the intent of section 5.4 to help laboratories better select, validate, and manage their methods, procedures, and instructions. We’ll also look at uncertainty of measurement and how it relates to test or calibration measurements and data control.

The majority of noncompliances found by the Laboratory Accreditation Bureau were based on assessment data collected since 2012. The noncompliances were likely due, in part, to confusion caused by the terms “method” and “procedure,” which are used interchangeably in the standard, whereas in practice they are not.

What are methods and procedures?

According to ISO/IEC 17025, clause 5.4.1, laboratories must “have and use appropriate methods and procedures.” The conjunction “and” infers that methods and procedures are two separate nouns having different meanings; however, ISO/IEC 17025 uses the words interchangeably, which many find confusing. “Procedures” include sampling, handling, storage, transport, and preparation of items to be tested or calibrated. Where appropriate for analyzing test and calibration data, methods and procedures are needed for measurement uncertainty and the statistical techniques used.

The Merriam-Webster dictionary defines “method” as “a systematic procedure, technique, or mode of inquiry employed by or proper to a particular discipline or art.” “Procedure” is defined as “a particular way of accomplishing something or of acting.” We see that these two words, in the context of ISO/IEC 17025, have similar meanings and the same basic goal: to accomplish a task in a manner yielding consistent results.

Generally, a method is a commonly available published document that describes a measurement or testing technique, e.g., ASTM E18—“Standard test method for Rockwell hardness of metallic materials” or ASTM E4—“Force verification of testing machines.”

Procedures can also be published documents that generally describe a number of steps to complete a task with specific outcomes, e.g., ASTM E18, Annex A—“Direct verification of Rockwell hardness machines,” or ASTM E4—“Force verification of testing machines by elastic calibration devices.”

Laboratories may find it useful to separate these ideas; however, laboratories would most benefit from arriving at their own understanding and use of the terms “method” and “procedure.”

The word “instruction” is used differently. It describes documents intended to ensure consistency. Among these are instructions for operating equipment, handling and preparing items to be tested or calibrated, and for any activity where the absence of instructions would jeopardize test and calibration results.

Methods, procedures, and instructions must be kept up to date (i.e., controlled) and available to the personnel for whom they are intended.

Allowable deviations from these can occur only under certain “documented” circumstances and the subsequent acceptance of them by the customer.

Selecting methods

Laboratories are required to use methods or procedures that meet customer needs and are appropriate for the test or calibration. This is typically determined during the contract review; to meet a need, the laboratory must first understand what that need is.

In clause 5.4.2, ISO/IEC 17025 states that a laboratory “shall preferably use international/regional/national published standards.” The word “preferably” makes this an optional “shall.”

A laboratory “shall ensure use of latest valid edition of standard (method/procedure) unless not appropriate or possible.” The word “unless” makes this a conditional requirement based on appropriateness, or an impossibility that should be well documented during the contract review.

The standard also states, “When necessary, a laboratory is required to supplement a method or procedure with additional details to ensure its consistent application.” The words “when necessary” make this a conditional requirement. If the method or procedure is sufficiently detailed, or laboratory personnel are sufficiently trained in making the application of the method or procedure consistent, then supplemental instructions may be unnecessary. The sufficiency or “fitness for purpose” of a method or procedure is determined by the laboratory and is typically confirmed through performance. In life science laboratories, this process is called “method validation” and is covered in clause 5.4.5. (See the “Method validation” section below.)

When a customer does not specify the method or procedure to be used, a laboratory “shall select an appropriate method/procedure.” Ideally, customers will know exactly what they require, but often they rely on the laboratory to have access to a subject matter expert. The requirement to understand and meet the customer’s need remains.

A customer “shall be informed as to the method or procedure chosen where the customer has not specified the method/procedure.” Again, the laboratory has a responsibility to first inquire about the customer’s need and then, if possible, recommend a method or procedure to meet that need.

A laboratory is required “to confirm that it can properly operate a method.” Where a published method or procedure is introduced, the laboratory must first confirm that it can accomplish the desired tasks. This normally need not be a full validation of the method or procedure, but rather a verification that the method can be properly executed.

A laboratory is required “to repeat such verification when a method, procedure, or instruction is changed, revised, or updated, or if the published reference method is amended, or if major equipment used in the method is changed.”

A laboratory must “inform the customer if a specified method or procedure is considered inappropriate or out of date, even if the customer specifies such an approach.” The laboratory must also confirm that it can perform the out-of-date method or procedure. This confirmation should be well documented as part of the laboratory’s contract review.

Laboratory-developed methods

In clause 5.4.3, ISO/IEC 17025 states, “A laboratory may, for its own use, develop methods or procedures. This must be a planned activity.” This is not a typical activity for a commercial laboratory. Normally, this is in support of or in the use of its own equipment design, or to support research and development in a new or different technique. This activity must be assigned to qualified personnel; typically, personnel are assigned specific tasks, equipment, time, and other resources.

Normal method development involves changes over time, and goals must adapt to these changes and challenges as well as to any discoveries made. Good communication is instrumental in preventing duplication of activities, sharing discoveries, and surmounting challenges.

Nonstandard methods

Clause 5.4.4 states, “When it is necessary to use methods not published as standard methods, this must be subject to agreement with the customer.” Where the use of a non-standard method is either because of a customer request or at the laboratory’s behest, customer approval of the nonstandard method is still required.

Clause 5.4.4 also states, “There must be a clear, documented specification of customer requirements and purpose for a test or calibration, and the method shall have been validated appropriately before use.” A clear set of requirements and well-defined purpose, backed by performance data, is necessary to determine a method’s fitness for use.

Method validation

This is defined as the confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled. To confirm that the method is fit for intended use, per clause 5.4.5, a laboratory “shall validate non-standard methods, laboratory developed methods, standard methods used outside their intended scope, and any amplifications (supplements or instructions) or modifications of standard methods.”


Figure 1: Method validation in a laboratory

Method validation techniques vary by scientific discipline, and these techniques are dependent on the scientific discipline in use.

For standard methods, a laboratory should normally verify its fitness for purpose. This process is usually less involved than full validation, but it might involve many of the same considerations.

Method instructions provide the detail necessary to ensure that all personnel involved are executing a method or procedure in a consistent manner, specific to the laboratory’s equipment, environment, data collection methods, and training programs. These factors may be included in the validation of the method to ensure they are consistent with the method’s intent and purpose.

Per clause 5.4.5, validation “shall be as extensive as necessary to meet the needs of the given application or field of application.” The words “as necessary” introduce some needed ambiguity because ISO/IEC 17025 is a “general requirements” standard; the degree of rigor is determined by the laboratory and is typically confirmed through performance after training. A method similar to an existing one in use may require little effort to validate, while a newly introduced method might require extensive training, multiple studies, or proficiency tests. A standard method introduced to a laboratory may require less effort (i.e., verification) to determine its fitness for purpose, compared to a nonstandard or laboratory-developed method that needs complete method validation.

Clause 5.4.5 also states, “A laboratory must document the procedures used and data collected.” After completing the validation work, the laboratory must also formally state whether it believes that the method or procedure is fit for its intended use. The range and accuracy of values obtained from the validated method as assessed for intended use “shall be relevant to customer’s need.” The data and information recorded must also be relevant to the customer’s need.

Uncertainty of measurement

Uncertainty of measurement is one of the parameters that a laboratory can use to determine the fitness for purpose of its tests and calibrations. Uncertainties of measurement are important performance parameters of a method. ISO/IEC 17025’s clause 5.4.6 requires testing and calibration laboratories to have (and apply) a procedure for estimating uncertainty of measurement for all calibrations and test measurements.

The simplest description of uncertainty is that it describes the confidence region, around the resultant value, that contains the results taken from a set of replicate measurements. Figure 2 demonstrates normally distributed results about the reported value.


Figure 2: Typical uncertainty described in the Guides to the expression of uncertainty in measurement (GUM series)

Where methods preclude rigorous and valid uncertainty estimations, laboratories shall, per clause 5.4.6, attempt (at least) to identify contributors and make a reasonable estimation, and they must ensure that their form of reporting results does not a give wrong impression of uncertainty. Great care must be given as to how uncertainty of measurement is reported.

A reasonable estimation of uncertainty must be based on knowledge of the performance of the method. This must be personal or organizational knowledge; using someone else’s knowledge is not adequate. The laboratory is required to also make use of (personal or organizational) previous experience and validation data. Unrelated experience or data are not considered adequate for this effort.

When estimating uncertainty of measurement, all contributors of importance must be taken into account, using appropriate methods of analysis. The laboratory’s procedure should determine the level of significance deemed important. Typically, it is acceptable to remove from consideration those contributors contributing less than 5 percent of the standard (k = 1) uncertainty. It is customary to include documentation in an uncertainty budget to describe the contributor and justify the omission of its contributing value. It is imperative that the analysis of each contributor is appropriate, and that the analysis considers the probability density function (PDF), its distribution and accompanying divisor, its sensitivity compared to other contributors, and any correlation to other contributors.

The procedure should instruct technical personnel regarding the minimum content of an uncertainty estimate, commonly called an “uncertainty budget,” the typical PDF associated with each identified contributor, and the review and approval processes for these budgets. Many uncertainty procedures describe the PDFs encountered with pictorial representations and identify contributors that typically exhibit these conditions. Typically, the laboratory will have a number of budgets associated with its scope of accreditation, which require the same control as any other document in the quality management system as well as the same frequency or conditions prompting their review.

The approval process for uncertainty estimates should include methods to ensure that all significant contributors have been identified; contributors that are not to be included should be accompanied by the technical justifications for their exclusion. Many laboratories employ a “sanity check” against examples provided in consensus standards or technical reports, similar laboratories’ scopes, National Metrology Institute (NMI) capabilities, or the International Bureau of Weights and Measures (BIPM) key comparison database to detect the possibility of an incomplete analysis.

Laboratories often have at least the same quantity of “working” budgets for daily laboratory use which technical personnel can manipulate to match the specific conditions (i.e., short-term stability, repeatability, environmental factors, etc.) or characteristics (i.e., instrument resolution, uncompensated bias, etc.) of the item being tested or calibrated. There should be sufficient controls in place to prevent inadvertent underestimation of uncertainty or reporting an uncertainty numerically lower than the laboratory’s scope of accreditation.

Control of data

Per clause 5.4.7, calculations and data transfers must be subject to appropriate checks in a systematic manner. This could be a data transfer from a paper worksheet to a computer program used to generate a report; a machine output to paper report; a machine output to computer program; a transfer of data from one computer program to another; or a data transfer between computers. Any transfer must be subjected to systematic checks appropriate to the transfer. When computers or automated equipment are used, the laboratory must ensure that the software is documented and validated. Where a laboratory develops its own or modifies purchased software, that software must also be validated as fit for use.

Clause 5.4.7 also states that procedures for protecting data “must include, but aren’t limited to, the integrity and confidentiality of data entry or collection, data storage, data transmission, and data processing.” The laboratory’s product is based on these data, and they must be protected. The equipment collecting, processing, transmitting, and retaining data must also be protected.

Conclusion

It is imperative that laboratories have a process in place for selecting methods, procedures, and instructions, however these are understood or defined. These documents must be evaluated (or validated) as being fit for the laboratory’s use. Laboratories that validate their own methods must have a clear understanding of the particular scientific discipline and their validation processes. Laboratories must also understand their measurement processes well enough to estimate their uncertainties of measurement. Laboratories should be aware that accreditation bodies and assessors usually will evaluate uncertainty budgets during their document review by determining if they can model the measurement process based on the information given in the submitted uncertainty budget. Where little information is given, or the information lacks sufficient description, more time is planned to review both the procedure and resultant budgets. Laboratories must also have procedures to control their data and any actions taken on that data because the data influence other measurements and data reporting.

Discuss

About The Author

Randy Long’s picture

Randy Long

Randy Long works as a calibration program manager at the Laboratory Accreditation Bureau. His duties include assessment team allocation, assessment package review, assessor evaluation, assessor training, laboratory assessment, and participating in the Technical Advisory Group (TAG). Previously, Long worked at Richard J. Bagan Inc. as a senior calibration technician and laboratory manager. He was instrumental in helping the company achieve ISO/IEC 17025 accreditation by writing a large portion the company’s ISO 9001 and ISO 17025 (Guide 25) quality systems, as well as the majority of the calibration procedures and measurement uncertainties.