Hannah Murfet (CQP MCQI, BSc, DipQ) Regulatory and Quality Manager, Horizon Discovery
This article aims to provide an overview of the clinical application of NGS and the implication of the recently released NY State Guidelines. A summary of the validation requirements and use of reference materials is below:
|Reference Standard/Control Use:|
Next Generation sequencing is becoming an appealing option with much higher throughput and potential to change the face of genetic medicine (American College of Medical Genetics and Genomics, 2013). However care must be taken to ensure high levels of control are maintained, this can be achieved through validation and reporting controls where specific NGS requirements are expected, in addition to those mandated by clinical accreditation and certification programmes.
|Clinical Context of NGS|
|Figure 1. Overview of the Clinical Context of NGS|
|Figure 2. Overview of Clinical NGS Requirement areas
In the clinical context quality assurance consists of the maintenance of a desired level of quality for laboratory services. Systematic implementation of a quality management system helps maintain consistency and quality of services offered, whilst allowing the development of efficiency and definition of authority.
Quality assurance is essential to ensure efficient and reproducible operations and document controls form one essential part of quality assurance. While the requirements specifically identify Standard Operating Procedures (SOPs) as a requirement, supporting documentation will help provide context.
Typically quality management systems take a three tier hierarchy. At the highest level the policies define the organisation’s strategy and focus; here you will find organisational goals, expectations and commitments. Procedures define and document instructions on performing business/quality management or technical activities. In the case of New York State Department of Health guidelines, there is clear focus on the requirement for SOPs.
|Figure 3. Document hierarchy for management system documents.|
SOPs can be broken into two levels. The first level represents the flow of information and useful for demonstrating the sequence of event and associated responsibilities or authorities. The first level procedures are best kept at a relatively high level and work well as flow diagrams supported by supplementing text such as the example below.
Figure 4. Level one process example including steps, assigned responsibility/authority and sub steps. A level one process may reference more specific and detailed level two processes.
An overview of the typical testing sequence is shown below; this may be incorporated into one or a few level one processes depending on the complexity of the clinical laboratory’s operations.
|Figure 5. Testing sequence overview.
Level two processes are best documented as clear ‘how to guides’, detailing all the responsibilities, materials and procedures necessary to complete the activity. For laboratory focussed activities, these are best used as inputs to and outputs from validation studies to establish consistent and clear protocols that support good training and operation of the laboratory.
Records are an essential practice and should include which test instruments were used in each test and documentation of all reagent lot numbers. Any deviations from standard procedures should be recorded, including any corrective measures to report and resolve any deviations. Templates may be generated to ensure consistency in output records for both testing and reporting.
In addition to documented processes, implementation of predetermined checkpoints or key performance indicators should be included to permit the monitoring of quality assurance over time. Once established these may act as a trigger for assay drift, operator variability or equipment issues.
Controls for reports and data must be implemented. In the US compliance to the HIPAA Act (Health Insurance Portability and Accountability Act) must be implemented to ensure traceability and protection of patient data. Many authorities mandate record retention periods, including CLIA who mandate that records and test reports must be stored for at least 2 years. With strong implementation of quality assurance systems clinical laboratories can assure reliability and continuity of results and may look to further certification such as the implementation of ISO 15189, especially in countries where there are no formal accreditation schemes.
Validation for clinical laboratories involves the assessment of various parameters including those used for protocols, tests, materials and platforms. The aim of validation is to provide confidence to meet critical requirements. The following data section (section 3) should be considered when determining the scope of validation required.
Figure 6. Validation and assurance levels.
An overview of the specific validation requirements for Next Generation Sequencing is given below, this may be used as a basic checklist of coverage, supplemental to more general accreditation or certification requirements e.g. those required by CLIA or ISO 15189.
Figure 7. Overview of NGS Specific Validation Requirements.
Given the huge amount of data that may be generated through Next Generation Sequencing, accurate and efficient systems are essential. Data is generally established through validation and then monitored through the establishment of predetermined checkpoints or key performance indicators to ensure consistency and accuracy of service provision.
Table 1: Overview of NGS Specific Data Requirements
|Validation including a minimum of 50 patient samples with representation for material type (FFPE) and variants across target areas and confirmed by an independent reference method. Minimum of 10 positive samples for each type of variant.
A recomended approach is to sequence a well characterised reference sample to determine specificity.
If vigorous validation of reported variants has not been completed in the original studies, ongoing confirmation by independent reference methods must be performed until at least 10 reference points have been independently validated.
A disclaimer must be used where incidental finding of unknown significance are included where there is no established confirmatory assay. The disclaimer must clearly state that the variant has not been verified.
|Robustness is the likelihood of assay success; adequate quality control measures must be in place to determine success of techniques such as extraction, library preparation or sequencing.|
|Precision is related to within run control.
For each type of variant a minimum of 3 positive samples containing variants near the stated sensitivity of the assay must be analysed in triplicate in the same run using different barcodes.
Renewable reference samples can be used to determine the analytical validity of the test. These can establish baseline data to which future modifications can be compared.
|Repeatability and Reproducibility|
|Repeatability and reproducibility is related to between run controls to determine ability to return identical results under identical (repeatability) or changed (reproducibility) conditions.
For each type of variant a minimum of 3 positive samples containing variants near the stated sensitivity of the assay must be analysed in three separate runs using different barcodes on different days by two different technologists where possible.
If multiplexing samples with distinct barcodes, it must be verified that there is no cross talk and that all target areas and variants are reproducible independent of which patient/barcode combination is used.
It is useful to consider instrument-instrument variability as well as interoperator variability. Parameters for expected reproducibility should be established, typically around 95-98%.
|Analytical Sensitivity and Specificity|
|Sensitivity and specificity refers to positive and negative percent respectively, in agreement of results when compared to gold standard.
Interrogate all types of variants in three target areas with consistently poor coverage and three with consistently good coverage. This can be established with defined mixtures of cell line DNA and not plasmids, but needs to be verified with 3 – 5 patient samples.
The limit of detection should be established.
Confidence intervals for variant types must be determined.
A minimum data set is expected to establish key performance characteristics
Figure 8. NGS minimum data set
|Table 2: Overview of Quality Controls|
In contrast to quality assurance where the infrastructure for quality is established to maintain the right service, quality control addresses testing and sampling to confirm outputs against requirements. Quality control takes place across various areas from reagents used, software and in assay controls:
Figure 9: Control Requirements.
Several QC requirements may be required and documentation of these requirements is best detailed in the relevant SOP to ensure they are documented and available at point of use. The quality control measures applied can vary depending of chosen methods and sequencing instrument but should include methods to identify sample preparation failures and failed sequencing runs.
Both accreditation programmes such as CLIA and standards certifications such as ISO 15189 contain specific requirements around the generation, approval, issue and re-issue of reports. The most essential requirements related to Next Generation Sequencing have been listed below.
|Table 3: Overview of Quality Controls|
The overall picture for Next Generation Sequencing is evolving with clearer requirements for everything from quality assurance to reporting requirements. The key with Next Generation Sequencing, as with any clinical technology is to establish validity and clearly document any assumptions or limitations of the platforms and assays involved, especially when reporting to the requesting clinician. While there is not complete understanding the clinical implications of some variants, there are clear prospects emerging for this technology to improve the standard of care and further advance the development and clinical support for companion diagnostics.
Horizon Diagnostics is proud to support clinical laboratories with the provision of sustainable reference materials for research use; applications include the validation of equipment and consumables offline from patient testing. This report has been provided for information only.
American College of Medical Genetics and Genomics. (2013, September).
New York State Department of Health. (2014, January).
If you’re having trouble finding what you’re looking for or require more information on this topic, please Contact Us.