Verification or validation, that is the question
Perspective

Verification or validation, that is the question

Giorgia Antonelli1, Andrea Padoan1, Ada Aita1, Laura Sciacovelli2, Mario Plebani1,2

1U.O.C. of Laboratory Medicine, Department of Medicine, University of Padova, Padova, Italy; 2Department of Laboratory Medicine, University-Hospital of Padova, Padova, Italy

Correspondence to: Giorgia Antonelli. U.O.C. of Laboratory Medicine, Department of Medicine, University of Padova, via Giustiniani 2, 35128 Padova, Italy. Email: giorgia.antonelli@unipd.it.

Abstract: The latest revision of ISO 15189 standard in 2012 is now universally considered as the most important document for accreditation of medical laboratories. This standard is aimed to verify and validate examination procedures (EPs). Verification is “provision of objective evidence that a given item fulfils specified requirements”, while validation is “verification, where the specified requirements are adequate for the intended use”. When used without modification, validated EPs shall be verified, whilst non-standard methods, home brew methods, validated methods which have been modified or are being used outside their intended scope shall be validated before being introduced into clinical and laboratory practice. In order to fulfil the above requirements, three different approaches were used for: (I) verification of EPs already in use in laboratory; (II) verification of newly introduced EPs; (III) validation of EPs. For verification, at least two performance characteristics should be verified according to the intended use of specific EPs. Different operative flows are described for quantitative, qualitative and semi-quantitative methods. For validation procedure, a validation plan should be planned, including performance characteristics for specific EP, the experimental procedure and acceptability criterion. The final step of the validation process is production of a validation certificate. This is aimed to summarizing the validation process and defining the approval for being used in clinical practice. The proposed verification and validation procedures were actually fulfilling the ISO 15189 requirements during the accreditation audit. They can hence represent a model to follow for other medical laboratories.

Keywords: ISO 15189; verification; validation; examination procedure (EP); accreditation


Received: 19 July 2017; Accepted: 20 July 2017; Published: 11 August 2017.

doi: 10.21037/jlpm.2017.07.11


ISO 15189:2012 requirements 5.5.1.2 and 5.5.1.3

The latest revision of ISO 15189 standard in 2012 is now universally considered the most important document for accreditation of medical laboratories (1), and 60 countries have made it mandatory for clinical laboratories so far (2). The standard focuses on patient care as the main objective of laboratory services, which can be achieved by verifying management organization, responsibilities and technical quality, and by adopting the philosophy of continuous improvement (3). The technical needs focus on mitigating laboratory errors through evaluation of the total testing process. Regarding the analytical phase, this standard entails verification and validation of examination procedures (EPs) (section 5.5.1.2 and 5.5.1.3). According to International Vocabulary of Metrology 3 (VIM 3), the definition of verification is “provision of objective evidence that a given item fulfils specified requirements” (4). In particular, the ISO 15189 states that “the independent verification by the laboratory shall confirm, through obtaining objective evidence (in the form of performance characteristics) that the performance claims for the EP have been met. The performance claims for the EP confirmed during the verification process shall be those relevant to the intended use of examination results.” (1). Validation is defined as “verification, where the specified requirements are adequate for the intended use” (4). In particular, ISO 15189 attests that “the validation shall be as extensive as is necessary and confirm, through the provision of objective evidence (in the form of performance characteristics), that the specific requirements for the intended use of examination have been fulfilled.” (1).

When used without modification, the validated EP shall be verified, whilst non-standard methods, home brew methods, validated methods which have been modified or are being used outside their intended scope shall be validated. The main objective of validation of an EP is to demonstrate its fitness-for-purpose (5).


The pragmatic approach to verify and validate the examination procedures (EPs)

In order to fulfil the above requirements, three different approaches were used for:

  • Verification of EPs already in use in laboratory;
  • Verification of newly introduced EPs;
  • Validation of EPs.

Verification of EPs already in use in laboratory

The developed operative flow for EPs already in use for over two years is based on consideration that the accreditation process is developed in a laboratory where the quality system has already been successfully implemented (6,7). Therefore, the operative procedures refer to a laboratory in which the total testing process is under control, users’ satisfaction is monitored and an improvement process is ongoing. Therefore, if the EP provides results satisfying clinical needs and, subsequently, the purpose of the test, the verification can be limited to parameters focused on the accuracy of test results. This approach is described in detail elsewhere (8), and summarized in Table 1.

Table 1
Table 1 Scheme of the procedure for the verification of examination procedures already in use by at least 2 years
Full table

Verification of newly introduced EPs

Following the same approach as for verification of EPs already in use, different procedures were identified for quantitative, qualitative and semi-quantitative newly-introduced EPs.

Verification of newly introduced quantitative EPs

For quantitative EPs, the document CLSI EP15-A3 (9) was analysed for establishing the most suitable workflow. As for quantitative EPs already used in the laboratory, no less than imprecision and trueness need to be verified in terms of CV% and bias%, respectively.

Imprecision verification

As suggested by CLSI EP15-A3 (9), the imprecision verification study consisted of three parts, as summarized in Figure 1.

Figure 1 Operative flow for imprecision verification of newly introduced examination procedures. SWL, within-laboratory imprecision; SR, laboratory repeatability; σWL, manufacturer total imprecision; σR, manufacturer intra-assay imprecision; UVLs, upper limits of verification; UVLWL, upper limit of verification for total imprecision; UVLR, upper limit of verification for intra-assay imprecision; CV%ref, clinically allowable limit of variability.
  • Repeated measurements over 5 days of at least two patients’ samples. For this study, patient samples, pool of patient samples or commercially available quality controls can be used, in this hierarchical order of priority. The samples should have different concentrations and in particular, the concentrations should be close to those validated by the manufacturer. The proposed basic design (3×5) needs five days, three replicates per run, one run per day, for a total of 15 results per sample. In relation to the measurand stability, the length of the design can be reduced to a 3×4 scheme, but not less than a 3×3 design should be used. The presence of outliers needs to be investigated using the Grubbs’ test. If more than one outlier is observed throughout the entire study, the verification of imprecision study should then be repeated.
  • Calculations of within-laboratory imprecision (SWL) and laboratory repeatability (intra-assay-SR).
  • Assessment of uniformity with claims and acceptability of test results. For each level, the SWL should be less than the total imprecision claimed by the manufacturer (σWL).When SWL is higher than σWL, the upper verification limit (UVLWL) needs to be calculated, as described in the CLSI EP15-A3 (9), by considering the degrees of freedom and the UVL factor. If the SWL is higher than the respective UVLWL, a clinically allowable limit of variability (CV%ref) needs to be defined. This limit can be identified following the hierarchical structure established in the 1999 Stockholm Consensus Conference (10) and revised in the 1st strategic conference held in Milan (11), according to: (i) clinical recommendations; (ii) biological variation; (iii) state-of-the-art, as we proposed for verification of EPs already used in the laboratory (8). If SWL is higher than CV%ref, the SR has to be compared with the intra-assay imprecision claimed by manufacturer (σR). If SR is higher than σR, the UVLR needs to calculated, as previously discussed. If the SR is higher than the respective UVLR, the verification process fails.

Trueness verification

Trueness is estimated by analyzing materials with known concentration, comparing results with target values, and establishing the verification interval, as described in the CLSI EP15-A3 document (9). The materials with known concentration can be: (I) materials whose concentrations can be adjusted to the desired levels, with negligible imprecision, e.g., by spiking a therapeutic drug into patient sample pools known to be analyte-free; (II) certified reference standards; (III) survey materials from PT/EQA programs; (IV) materials used in inter-laboratory quality control programs; (V) materials intended for routine internal quality control with formerly assigned target values. The mean value obtained in these studies should be included in the verification interval. If the verification interval did not include the mean value, the bias should be calculated (Bias%LAB). If the manufacturer provides a declared bias, the bias obtained by the laboratory should be less than or equal to that claimed by manufacturer. If the manufacturer did not declare this value, a specific allowable bias (Bias%ref) has to be identified, according to criteria reported for imprecision. If the Bias%LAB is higher than the Bias%ref, then the verification fails. When materials at known concentrations are unavailable, another performance characteristic should be verified according to performances claimed by the manufacturer, so that at least two performance characteristics are verified.

Verification of newly introduced qualitative EPs

For qualitative EPs, the document CLSI EP12 (12) is used as a guideline. For this type of procedures, the diagnostic accuracy is evaluated, in terms of sensitivity and specificity, using not less than ten true negative samples and not less than ten true positive patient samples. To verify this parameter, the manufacturer needs to declare the diagnostic accuracy in terms of diagnostic sensitivity, diagnostic specificity, number of true positive and true negative samples. The 95% confidence intervals (95% CI) are calculated with the Wilson’s method, as we proposed for verification of EPs already used in laboratory (8). The diagnostic sensitivity and specificity obtained by the local laboratory should be included in the calculated 95% CI derived from manufacturer’s data.

Verification of newly introduced semi-quantitative EPs

For semi-quantitative EPs, imprecision and diagnostic accuracy are assessed, as described above for quantitative and qualitative EPs, respectively.

Validation of EPs

The validation process consists of six parts:

  • Analysis of available scientific documentation pertinent to specific EP. In particular, clinical guidelines, recommendations and peer-reviewed texts or journals should be considered. In our experience, documents are selected for validation of mass spectrometry-based methods (13,14), for chromatographic techniques (15,16), for ligand-binding assays (15), for cell-based fluorescence assay (17), and for multiplex nucleic acid assays (18).
  • Evaluation of the intended use of EP.
  • Identification of performance characteristics for specific EP. The choice of the performance characteristics is strictly dependent on the necessity to confirm that specific requirements for intended use are fulfilled, as mandated by the ISO 15189 (1).
  • Definition of experimental procedure: the type of the tests to be performed, thus including number of samples and replicates, along with the time need for testing should be determined.
  • Identification of acceptability criterion for evaluating results according to the appropriateness of intended use. The acceptability of results should be evaluated according to reliable scientific documents, previously identified.
    A validation plan should be planned, summarizing all these points. The template of a validation plan is briefly shown in Figure 2. All results obtained during the validation process should be recorded, along with instrumental output. In fact, the ISO 15189 states that “the laboratory shall document the procedure used for the validation and record the results obtained” (1).
  • Production of a validation certificate, i.e., the final step of the validation process. The validation certificate is a tool aimed to summarize the validation process and define the approval for use in clinical practice. The validation certificate should report type of EP, intended use of the test, reference documents, results obtained during the evaluation of performance characteristics and approval for use. In particular, the personnel who carried out the validation and who is responsible for specific EP has the duty of the fitness for purpose of validating EP. The Laboratory Director has the responsibility for adopting and using the EP in clinical practice. The template of a validation certificate is shown in Figure 3.
Figure 2 Template of a validation plan.
Figure 3 Template of a validation certificate.

Conclusions

The International Standard ISO 15189 is widely recognized as the guideline for medical laboratories accreditation. Despite the first edition was published in 2007, this guideline has not been extensively implemented so far by medical laboratories at an international level, with large variations between different Countries and Regions. For example, there are only six accredited laboratories in Italy (19). The verification and validation of the EPs are novel requirements compared to the more widespread ISO 9001 certification. The development of operating procedures based on a practice approach should help medical laboratories implementing the accreditation process, thus considering the balance between technological possibilities, risks and costs. The proposed verification and validation procedures have been recognized to comply with the ISO 15189 requirements during the accreditation audit. Therefore they should represent a valuable model to be followed by other medical laboratories.


Acknowledgements

None.


Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.


References

  1. International Organization for Standardization (ISO). Medical laboratories-Particular requirements for quality and competence. ISO 15189:2012.
  2. Farmer T. Toward a culture shift in laboratory quality: application of the full ISO 15189 standard. MLO Med Lab Obs 2015;47:38-9. [PubMed]
  3. Plebani M, Sciacovelli L, Chiozza ML, et al. Once upon a time: a tale of ISO 15189 accreditation. Clin Chem Lab Med 2015;53:1127-9. [Crossref] [PubMed]
  4. International vocabulary of metrology-Basic and general concepts and associated terms (VIM) JCGM 200:2012. VIM 3rd edition, Geneva, 2012.
  5. Stöckl D, D'Hondt H, Thienpont LM. Method validation across the disciplines--critical investigation of major validation criteria and associated experimental protocols. J Chromatogr B Analyt Technol Biomed Life Sci 2009;877:2180-90. [Crossref] [PubMed]
  6. Clinical Pathology Accreditation (UK). Standards for Medical Laboratory. Version 2.02 (March 2009).
  7. International Organization for Standardization (ISO). Quality management system. ISO 9001:2008.
  8. Antonelli G, Padoan A, Aita A, et al. Verification of examination procedures in clinical laboratory for imprecision, trueness and diagnostic accuracy according to ISO 15189:2012: a pragmatic approach. Clin Chem Lab Med 2017. [Epub ahead of print]. [PubMed]
  9. Clinical and Laboratory Standards Institute (CLSI). User Verification of Precision and Estimation of Bias; Approved Guideline-Third Edition. CLSI EP15, 2014.
  10. Fraser CG. The 1999 Stockholm Consensus Conference on quality specifications in laboratory medicine. Clin Chem Lab Med 2015;53:837-40. [Crossref] [PubMed]
  11. Defining analytical performance specifications: Consensus Statement from the 1st Strategic Conference of the Sandberg S, Fraser CG, Horvath AR, et al. European Federation of Clinical Chemistry and Laboratory Medicine. Clin Chem Lab Med 2015;53:833-5.
  12. Clinical and Laboratory Standards Institute (CLSI). User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline-Second Edition. CLSI EP12, 2008.
  13. Clinical and Laboratory Standards Institute (CLSI). Mass spectrometry in the clinical laboratory: general principles and guidance; proposed guideline. CLSI C50-P, 2007.
  14. Clinical and Laboratory Standards Institute (CLSI). Liquid chromatography-mass spectrometry methods; approved guideline. CLSI C62-A, 2014.
  15. Food and Drug Administration. Bioanalytical method validation. FDA, 2013.
  16. European Medicines Agency. Guideline on bioanalytical method validation. EMA, 2012.
  17. Wood B, Jevremovic D, Béné MC, et al. Validation of cell-based fluorescence assays: practice guidelines from the ICSH and ICCS - part V - assay performance criteria. Cytometry B Clin Cytom 2013;84:315-23. [Crossref] [PubMed]
  18. Clinical and Laboratory Standards Institute (CLSI). Verification and validation of multiplex nucleic acid assays; approved guideline. CLSI MM17-A, 2008.
  19. Accredia [Internet]: accredited Italian medical laboratories. Available online: http://www.accredia.it/accredia_labsearch.jsp?ID_LINK=293&area=7
doi: 10.21037/jlpm.2017.07.11
Cite this article as: Antonelli G, Padoan A, Aita A, Sciacovelli L, Plebani M. Verification or validation, that is the question. J Lab Precis Med 2017;2:58.

Refbacks

  • There are currently no refbacks.