Hello, my name is Mark Ewaldt. I'm a hematopathologist and molecular genetic pathologist at the University of Colorado Anschutz Medical Campus. I'll be talking to you today about approaches to design and validation of next generation sequencing assays. I have no financial disclosures to report. When we think about any clinical test, we think about a complex life cycle that this test goes through. It begins with test development and test design, which is one of the main focuses of our talk today. We'll discuss this in more detail in the coming slides, as well as test validation, where we ensure that the performance of the test we've designed is optimal for the clinical scenarios in which we are hoping to deploy this assay. And lastly, we won't spend a lot of time during today's talk about this, but quality management ensures the ongoing accurate performance of this assay, both through ongoing run level quality control with the presence of positive, negative, and no template controls, and by enrollment in routine proficiency testing, which is often managed through the College of American Pathologists. Although depending on the specifics of an assay, there may not be commercial PT available, and therefore alternate PT can also be engaged. When we think about developing a new test, the design considerations often reflect many of the clinical questions that are going to be asked by this test. For the sample type, we often wonder, are there specific matrix considerations? If we're looking for a germline test, we tend to work in blood or in buccal swabs, and we think about the matrix considerations of those samples, whereas tumor testing is often performed on formalin-fixed paraffin-embedded tissues, and this raises a whole separate set of matrix considerations. As we're designing our assay, we want to think about what are the particular variants that must be detected for this assay? Are they just single nucleotide variants? Are they copy number variants? And this is related to the specific clinical scenario and what the known landscape of alterations is in the set of diseases being assessed. Next, the sensitivity of the assay is critically important, and it's important to understand what level of variant allele fraction is relevant clinically in your sample population. In germline testing, it's often sufficient to look at variant allele fractions of 0, 50, and 100%, assuming that there's no mosaicism. However, in somatic testing for cancer, it's often important to look at much lower variant allele fraction. And then some of the workflow issues that relate to the sample throughput in your lab and the required turnaround time of your assay relate to multiplexing for how many samples are planned per run, and for turnaround time in terms of answering how urgently are the results needed. Some assays may not require particularly fast turnaround time, and a result of two to three weeks is perfectly fine clinically. It doesn't lead to a delay in care for that patient, whereas other assays, a turnaround time of two to three weeks may lead to delay in treatment with a particular agent and lead to significant harm for your patient. Fortunately, there are many guidelines papers that are generated by many of the specific professional societies, including the ACMG, the Association of Molecular Pathology, or AMP, and the College of American Pathologists, or CAP. This particular article that I'm citing here was published in 2019, and although it focused on the design and implementation of next-generation sequencing testing for inherited disorders, many of the principles can be extended to designing assays for oncology as well. This manuscript provides more of a framework rather than a long descriptive explanation for how to develop this, and it has a relatively short description in the manuscript, and then it comes with several attached worksheets in Microsoft Excel format that walks the reader through every stage of the next-generation sequencing lifecycle, from design to optimization, validation, test launch, and quality management. And each of these worksheets has multiple tabs with multiple templated regions that the reader can download and make alterations to ensure that they fully consider all the aspects of their assay design and NGS assay lifecycle. But thinking about it more generally and how we think about it for all of our assays, all next-generation sequencing assays consist of specific and recurring themes of particular stages. A sample is received in a laboratory, and it is prepared, and nucleic acid is extracted. This nucleic acid then goes through a process called library preparation, in which the regions of interest or the areas that are the targets of sequencing are enriched, and patient and platform-specific barcodes are attached to these molecules of DNA that will be sequenced. Then these are loaded onto a next-generation sequencing instrument for massively parallel sequencing, and then there's subsequent analysis, both through automated analysis and bioinformatic workflows, as well as occasionally automated analysis for clinical interpretation, and in many cases manual analysis by a variant annotation scientist, a clinical geneticist, or a molecular genetic pathologist. As I mentioned before, one of the main clinical questions is what are the different samples that are going to be tested, and really different samples raise different potential challenges. One of the easiest sample types to work with is generally fresh tissues, because these have intact nucleic acid. So these are things like peripheral blood, saliva, buccal swabs, and really fresh frozen or snap frozen tumor samples, or even fresh tumor samples. These tend to have very high quality DNA with a lot of large, high molecular weight DNA molecules, and they tend to suffer from very few DNA damaging artifacts. Unfortunately, in oncology testing, most of the samples that are clinically available are formalin-fixed paraffin-embedded tissues, and this fixation process leads to various types of damages to the nucleic acid. It can lead to deamination and depurination of the nucleic acid bases, and it also leads to fragmentation. So the majority of the DNA retrieved from FFPE is much smaller in size than that from fresh tissues. In addition, particularly in oncology, it's important to think about the cellularity of your sample. As I mentioned before, with germline testing, usually we think about samples that are either homozygous, heterozygous, or absent for particular alteration. In tumors, we have to think about both the neoplastic cell population and the non-neoplastic background population, and this includes things like stromal cells that are adjacent to the tumor and supporting the tumor, as well as an inflammatory background, as some tumors incite a very strong lymphocytic or neutrophilic inflammatory response. The next step after we get through sample preparation is the library preparation process, and generally speaking, there are three different ways to go about the initial steps of library preparation. There is a PCR or amplicon-based library preparation in which a set of forward and reverse PCR primers are designed around a region of interest that will be amplified, and then these amplified fragments, which have been captured, can then go through a library generation process where they have adapter ligation added on, although the adapter ligation may also be part of the PCR primers. The other major approach that we think about is hybridization-based capture, and in this, the genomic DNA is fragmented, it undergoes end repair and A-tailing, and then the specific adapters for a particular patient and particular sequencing platform are attached to these DNA molecules, and then these molecules undergo amplification, and the amplified fragments are then subject to hybridization capture with particular baits to enrich for the target sequences. The third way of generating a library is somewhat simpler than either of these other methods, because it's for whole genome sequencing, and there's no need to select for a particular region of interest, as the whole genome is the region of interest. And so here, the genomic DNA is just brought through library generation, again that fragmentation, end repair, and adapter ligation, and then it undergoes amplification. And the output of all of this is a set of nucleic acid molecules that have patient and platform-specific barcodes on the ends and correspond to the regions of interest for the sequencing. When we think about which particular approach is best in which particular setting, obviously whole genome is in its own class for doing whole genome sequencing, but for hybridization-based sequencing, we definitely need to deploy this type of methodology for large panels, like we would see in whole exome sequencing, as this tends to avoid some of the PCR-based artifacts that can be seen in amplicon-based sequencing. Amplicon-based sequencing, however, tends to be more rapid than hybridization-based sequencing, so for a relatively small panel, where it's less concerning to have potential PCR jackpotting artifacts, or there's less concern for the interference of the many different multiplex PCR probes, PCR-based solutions may be the best choice. In addition to deciding which particular library preparation method that's going to be utilized, it's also important to think about the particular variant types that are going to be detected. All of these particular library preparation methods are well-suited to detecting single nucleotide variants and small indels, but larger insertion and deletion variants and chromosomal rearrangements or structural rearrangements can be challenging with simple designs of library preparation and may require more complex approaches. Some of these approaches include doing sequencing of large regions of non-coding areas, which is where most structural rearrangements occur. Also doing capture-based methodologies or anchored multiplex sequencing, which is very similar to 5' race, where a specific target is amplified from one end and a universal or nonspecific PCR primer is used at the other end to develop a targeted sequencing approach. Also, when considering copy number variation, it's important to do a large validation of normal samples to understand what the variability of the assay is and to correlate this to things like array-based testing, which are well-characterized ways of determining copy number variation. The assay sensitivity that's desired and can be delivered by your assay is going to impact both the depth of sequencing at a region and the multiplexing ability that can be done in your laboratory. As I've mentioned, germline samples tend to be expected to be 100% pure samples, and we tend to be looking for homozygous, heterozygous, and hemizygous mutations, and these would be present at 0, 50, and 100% variant allele fraction. To be able to sensitively detect at these different levels, it's generally accepted that a minimum depth of 30x or sequencing a particular position in the genome 30 times is sufficient to generate sensitive, high-quality data for all positions. In order to achieve a minimum depth of 30x, many assays are designed to have an average depth of 100x. When we think about how many samples can be multiplexed, the number of samples for multiplexing is defined really as the size of the run or the number of base pairs on the particular sequencing platform divided by the depth of sequencing and the panel size. So, based on these numbers with an average depth of 100, we would expect to have one particular number for the average multiplexing for a germline sample. Somatic and cancer sequencing, we expect that our samples are going to be less than 100% pure, and the required sensitivity of the assays can vary greatly based on what the input of tumor cells is. For many oncology assay applications, we set a 5% variant allele fraction as our lower limit of detection, and this is for standard diagnostic purposes, and in this case we would need to have 10% tumor cell purity in our sample in order to detect a VAF of 5% for a hemizygous or heterozygous variant. In order to achieve this much lower level of sensitivity, generally the minimum accepted depth is at least 250x at each position, and this leads to an average depth of 500 to 1000x. The way this impacts multiplexing, as you can see, to sensitively detect at the clinically required sensitivities, you can get one-fifth the number of samples multiplexed for a somatic or cancer assay as you could for a germline assay on the same sequencing platform. And last, one of the biggest concerns for many of our clinicians is the turnaround time that we're going to be providing them for sample in lab to report out, and one of the main areas where this can be slowed down is with the library preparation step. Whole genome sequencing is relatively quick because it does not require any type of enrichment step. There's no need for PCR-based amplification of regions of interest or hybridization capture. Amplicon-based methods tend to be relatively fast and rely on a multiplex PCR step, but this is not nearly as slow as capture-based methods, which have hybridization steps that range anywhere from 4 to 72 hours for the nucleic acid hybridization, and this leads to a much slower performance. The specifics of the particular sequencing platform also relate to the turnaround time, as ionometric sequencing platforms tend to have shorter runtimes than fluorometric sequencing platforms. And the informatics also can add a significant amount of time, as large panels with complex informatics being performed on samples with very high depth of sequencing are going to take more informatic processing power and can take longer to process. And I know that we've been speaking a lot in sort of abstract principles, and so I think a case is worth a thousand words. So your children's hospital wants to develop a panel for congenital heart disease. This test is currently a send-out test. It's performed on peripheral blood, covers 89 genes, and it's really focused on detection of heterozygous or homozygous variants with not much consideration for mosaicism, but it's a fairly low throughput assay or low sample expectation for your lab, only five samples per week, and it's not considered particularly clinically important critically, so these patients can get a turnaround time of two weeks, and this is considered acceptable. So when we go back to thinking about some of those principles we discussed earlier, this assay uses fresh samples, so it uses peripheral blood. It's important that while the sample that's being sent to the outside lab is peripheral blood-based, in your particular center there may be some benefits to allowing collection of saliva or buccal swabs, particularly if this test might be performed on infants where peripheral blood draws are more challenging to obtain. Also, it's important to think about whether or not there will be comparisons that are being performed here, either pairwise comparisons between the affected proband and a parent or a sibling, or trio-based comparisons, which typically correspond to testing of a proband with both parents, and this can allow for processing large amounts of data, as we'll see later. This assay size is 89 genes, and it's focused on coding variants, mostly SNVs and small indels, and because of that, it's really amenable to any type of library prep, either PCR-based or hybridization-based. Because the turnaround time is relatively generous with two weeks, there really isn't any pressure for PCR over hybrid capture. It's going to be dependent on the specific preference of the laboratory. But when we think about the overall test menu within a clinical laboratory, particularly in the area of germline testing, we want to think about, should we develop multiple targeted panels, or should we think about doing a single exome-based assay and finding bioinformatic ways to tailor the specific panels to the specific indications? So targeted panels can be helpful if you only have a few genes that you care about, you need a very high throughput, or you're concerned about overloading your informatics bandwidth. But if you expect to have many, many genes or an ever-changing gene list, having the flexibility of an exome-based panel is very helpful, as it's very easy to add clinically relevant genes to your validated list because you don't need to redesign the wet bench. In addition, as I mentioned, this allows you to deploy multiple different assays just by tailoring the specific genes from the exome that are being reported. Unfortunately, this will take longer to analyze through the informatics pipeline. It won't allow sequencing of as many samples on a run because the region being sequenced is so much larger, and it may be more expensive to do this if you only have a single indication that you really care about in your laboratory. I mentioned that we were going to talk about the use of trios, and here we have some trios with a couple of different alterations seen in this particular setting. Here we have the father, who's found to have a GATA4 alteration present at a VAF of 47%. The mother has a SIBL duplication present at 52%. Just based on the variant allele fractions, these look like they're heterozygous variants, and these are both present in genes that are associated with hereditary cardiac disease. When we look and sequence the daughter or the proband in this case, we find that each of these parental alterations are present at about a heterozygous variant allele fraction, but in addition, this patient has a novel variant, the SOS1 variant, and by having both of these unaffected healthy parents present, we're able to do this analysis and informatically remove the first two as unlikely to be contributing to the patient's phenotype. Switching gears to a different type of case, let's talk about if your cancer center is interested in developing a solid tumor hotspot assay with gene fusions. As I mentioned before, many of the tumor samples that come into clinical molecular diagnostics laboratories are from formalin-fixed paraffin-embedded tissues. The requirements for this assay is that it be able to cover 50 genes, manage possibly very low tumor cellularity, and allow for a lot of different variant types, including single nucleotide variants, insertion-deletion variants, splicing variants, and fusions, and it's expected that there will be about 10 samples per week, and the turnaround time here is a one-week turnaround time. So higher throughput and a little more clinically urgent to the ordering team. As I mentioned before, FFPE is a challenging matrix, so there are many studies that look at the specific changes that can happen in FFPE, and this particular paper from Jennings et al. was looking at how to optimize NGS sequencing and oncology, and both how to optimize it during assay design and how to monitor for ongoing performance during validation and into continuing quality control. So DNA extracted from FFPE may be highly fragmented, so it's important to optimize your extraction and design a library preparation method that's able to handle very small and fragmented DNA. For some of the deamination and depurination artifacts, it's possible to provide treatment to the nucleic acid to try to reverse some of these damages, or to do altered sequencing using duplex reads to try to suppress some of these errors, and we don't have time in today's conversation to discuss that, but it's something to consider. When we think about dealing with this, we have to worry about contamination, because rather than having a single needle for a peripheral bloodstick, we have these paraffin-embedded tissues, and they tend to get cut on our microtomes in the clinical laboratories, and it's important to change the blades between each tissue sectioning to prevent the contamination of nucleic acid from one patient to the next. There are also stochastic errors to be considered, amplification errors, and capture bias in hybrid capture-based assays, and then if you're dealing with PCR-based assays, it's important to consider primer bias and allele dropout when designing these assays. So as I mentioned before, FFPE samples can have variable tumor cellularity and nucleic acid quality. So this figure on the left is something from my current laboratory, where we have a melanoma sample. You can see it's these nests of pigmented cells shown here. Our laboratory employs the use of manual microdissection, and so by having a technologist go in with a very fine-tipped scalpel, they're able to remove just the neoplastic cells, which leads to very high tumor enrichment, and this allows us to be able to sequence samples that otherwise might have a very low overall tumor cellularity by enriching them to have a very high level of tumor cellularity in the sample for which we extract nucleic acid. On the right, I'm showing you the results from a tape station commercial assay. This is an assay that can be used for assessing nucleic acid size and also concentration, and here you can see that extracting nucleic acid from multiple different FFPE samples, there's a wide variety in the size distribution of the DNA fragments recovered after nucleic acid extraction. However, there tends to be a lot of smaller, few hundred base pair size fragments compared to several kilobase size fragments, as might be seen using a fresh tissue source. I mentioned that for this particular assay, there was a desire to detect rearrangements, and more complex variants are going to impact the specifics of the assay design. So, CNVs and indels tend to be focused around coding regions, and so if your assay only looks for these types of alterations, it's okay to design an assay that enriches just the exonic regions. When considering splice variants, the splice breakpoints frequently occur within introns, and so it may be important to sequence the entire, much larger intronic region, or it may be helpful to consider RNA-based testing. Here we have an example of met-exon-14 skip alterations. So, all of the alterations shown here are different types of nucleic acid variants that are detected in tumor samples of patients with lung cancer, and all of these lead to a variant splicing that leads to skipping of exon-14 of the met-oncogene. As you can see, these can be anywhere, starting deep within this intron to intron-14 as well. And when we think about going to RNA-based testing, I'm going to use here BCR-ABL as an example, we see that the exons, shown as these taller boxes, tend to be smaller than these large intronic regions that separate them. However, the breakpoints occur within these particular introns, and either you can sequence the entirety of the intron to detect the alteration, or by going to the RNA, performing reverse transcriptase, and getting the cDNA, we're able to sequence basically a nucleic acid molecule that has all of the intronic regions removed, but still lets us see if a variant splicing or a gene fusion event occurred. And as I mentioned earlier, if you have a need for increased sensitivity, it requires increased sequencing depth. Here I'm showing you a view of IGV, or the Integrated Genomics Viewer from the Broad Institute, and here we can see each of these red or blue lines corresponds to a read or a string of DNA sequence that is generated from your sequencing instrument at a particular position. The positions here correspond to the particular genomic coordinate, and here these gray bars reflect the depth of sequencing at that region. And we can see in particular here that we have very, we have fairly deep sequencing of the exonic regions, but relatively low level sequencing of the intron region here. And for this particular assay that we're talking about where we have a relatively small footprint of only 50 genes, but a relatively fast turnaround time with requiring one week turnaround for the assay, we may favor using a PCR based library approach. It tends to be faster than hybrid capture, and by not having too many genes, we're less concerned about the impact of of interference between our multiplex PCR steps. Now that we've given some examples for some of the considerations of how to design an assay, you're probably wondering how do I validate these tests? And again fortunately there are a variety of guidelines papers from the professional societies to help guide us. I've listed them here, and there, this is not an exhaustive list of all of the guidelines, but these are some of the major guidelines papers that are out in the community. And as we talked about that complex life cycle at the beginning of the talk, really the next focus is on that next step after test development into the test validation stage. And we want to make sure that our validation is able to cover several specific performance metrics, including sensitivity and specificity, the robustness of the assay, and the reproducibility. And to achieve these goals and make sure that our assay really is very robust, it's important that we consider all of the potential variant types that are going to be clinically important in the situations where this assay will be deployed, as well as the disease-specific aspects of this, and make sure that we cover some of the common pathogenic variants that we expect to see in our clinical samples. So it's important to think about some of these particular approaches to validation, and also think about some common organizing principles among all of those guidelines papers that I presented earlier. One of the main ideas that's present in these is an idea of an optimization and familiarization before the actual validation of the assay. This means running a small or medium-sized cohort of samples under the particular conditions that you anticipate to run your assay under, and then going through and analyzing your data and looking to see if there are particular areas where the assay can be optimized. This may relate to multiplexing of samples. If we expect a particular depth of sequence, and we anticipate that, say, we can get us a depth of 100x with five samples, and we find that we're at 80x with five samples for that particular assay, we may decide to decrease our run size to four samples being multiplexed, and repeat, and do this iterative process of optimization and familiarization with the performance of the assay before having a final lockdown assay that will be used for performance of the entire clinical validation process. Next, because next-generation sequencing involves assessing such a large portion of the genome with so many different alterations, and potentially the identification of novel variants, it's not going to be possible during validation to particularly test for every possible alteration that might be seen by the assay. Because of this, the concept of methods-based validation, or proving that your assay is able to work for your particular sample type and your particular variant type, has been introduced for next-generation sequencing, and has really taken off as the standard approach for validation of assays. This may be somewhat different, varying with state-to-state requirements, and in particular, New York State tends to have somewhat more stringent requirements for their validation, and may require more than just a simple methods-based validation, but that's not the scope of this talk, and so I would encourage you to look at the New York State Department of Health website if you tend to practice in New York or anticipate receiving samples from there. But in methods-based validation, we understand that not every variant is going to be possible to be tested, but we try to have multiple examples of each class of variant that we're testing, and we want to make sure that we in particular include specific common variants, and that we also look at hard-to-sequence areas. So we know that certain parts of the genome may have repetitive sequence, high GC content, we may, for whatever reason, get very low coverage of our assay in those areas. Those are the areas in particular, if there are pathogenic variants, that we really assess those in our validation and try to understand how our assay performs in that area. Next, we want to think about doing the wet lab validation, and this includes validation starting with the specimen type and specimen requirements, perhaps for an oncology assay you require that there's 10, 20, 30 percent tumor cellularity present, and you may require either fresh or formalin-fixed tissue, whereas for germline testing, perhaps you only want to deal with blood or you're willing to accept blood or buccal or salivary samples. Next, there are many different types of library prep, and it's important to think about which particular approach will be employed, what the requirements are, and how to assess the quality of the library generated, as well as looking at the performance of your sequencing run and how to assess the ongoing quality of that run. And after all this wet lab portion is done, a huge amount of data is being generated by the sequencing instrument, and there's going to be a bioinformatics validation, both using real data which is generated from the wet lab portion, but it's also possible to use in silico or synthetic data sets, and there's a large literature about different tools that exist to take a standard data file and create synthetic variants for analysis to test the performance of your particular pipeline for detecting different classes of variants. In addition, it's critical that as the bioinformatics software is updated, that there is strict version control, and that anytime an update is being deployed to the bioinformatics pipeline, consideration for repeating the bioinformatics validation with a mix of real and silico data is performed. So optimization and familiarization can really inform a lot about the sample and library quality metrics that you're going to generate from your samples. Again, this is tape station data, and we can see that we may assess that these two samples that are fairly smaller fragments are not going to perform well on our assay, and so we may decide during the optimization and familiarization step that samples that predominantly produce smaller fragments are not acceptable for our assay, and we will only accept samples that yield larger fragments shown here. In addition, we may use particular library quality metrics. Here we see another tape station. Only this is performed on the library that's generated for a particular sample, and we may decide that we will expect or require a particular size for our library that's being generated. If we don't see a particular size distribution, we may decide that a sample needs to undergo repeat library generation. And then looking at the particulars of the sequencing metrics, there are a variety of different types of metrics that are generated from a next-generation sequencing run. What I'm showing you here is data from a next-generation sequencing run on an Illumina-based platform, but we can see that we get for the particular reads that are being processed from each position on the instrument, we get the number of gigabases that are being sequenced. We get alignment rates. In this, we have Phi X, which is a bacterial genome, being spiked in as background, and we can see what the error rate is there. And one of the most important things that we look for is the Q30 or the quality score. And based on the validation and the familiarization, we may decide that we require a certain percent of our bases to have a quality score greater than Q30. Seen here, we have greater than 90% of our bases have a high quality score, but depending on the particular indication, it may not be required to have so many bases with a high quality score. In addition, as I mentioned, all of our samples during library preparation get patient and instrument-specific barcodes. So the instrument-specific barcodes in Illumina are the I7 and I5, and at those ends of the fragments, there are specific barcodes here that correspond to each of the indices or particular samples that are present. And then we can look and see how much of the sequencing was spread across these different samples. Here we see that we have somewhat limited variability in the amount of sequencing space that's being dedicated to each of the different samples, and this shows very good loading of our samples with uniform loading across. Here we see some more detailed information about the particular read where we can see the cluster density, which is an Illumina metric, for how many reads are seen per square millimeter of space on the physical flow cell. As I mentioned, there are some regions of the genome that we know are particularly difficult to sequence. This study looked at some of those particular areas that are difficult to sequence, and some of these difficulties may be related to homologous sequencing between multiple loci within the genome, or regions with repetitive sequence, or presence of pseudogenes. There is a lot of data about how to try to overcome some of these sequencing challenges, but it's important to address these hard-to-sequence areas during a validation process and to understand how your particular assay performs in these regions, and then to specifically determine if it's critically necessary if the assay needs to be redesigned to be able to achieve a higher performance at these regions, or if these regions should be excluded from final analysis and should be included in the methods and disclaimers portions of the clinical report to say that these particular areas cannot be confidently assessed by your assay. In addition, during that optimization and familiarization stage, it's possible to highlight areas where there's very poor coverage. Here we have data from a cancer hybrid capture assay. Here we see that we end up with our hybrid capture probe is probably somewhere underneath the highest portion of our bell curve, but we have less coverage of these regions of the exon, and we can see that it starts to drop off, and the minimum coverage of this particular exon drops to 2. So there's very little coverage out here at the end of the exon, and depending on what types of variants are present there, and if any of those variants might be clinically relevant, this may lead to some change to the assay design based on the optimization and familiarization. In addition, this stage of the validation effort can highlight some recurrent artifacts, and as you see here, we have a region of FOXL2, and this assay has very low coverage as shown up here with these small gray bars, and there tends to be a lot of noisy sequence where we see a lot of alterations. Similarly, here in ERBB4, we have a region that's a long stretch of A's. This is a homopolymer tract. Fortunately, it's within the intron, but we identify that there's both some insertion and deletion events, probably related to slippage during the library prep or sequencing steps, and these may be present in many of the samples seen in our validation runs. Based on comparison of the performance in these regions on the test being designed and an orthogonal assay can be determined if these are true variants or artifacts, and those that are recurrent artifacts seen in many, many samples can be specifically excluded from further analysis during your testing. In addition to many of the sample-specific metrics that come off the next-generation sequencing instrument itself, there are a variety of key informatics metrics that can be assessed during optimization and familiarization. Here I have a long list shown on the right for the validation of a malignant hematology-based probe set, but some of the important things to consider are the percent on target, also called percent-selected bases, so it lets you know how much of the sequencing region or the sequencing data is generated corresponds to the target region of interest for the assay that was designed. And then also it's important to think about coverage depth for the targets, thinking about things like mean, median, and minimum coverage, and to understand what kinds of performance you get at each of those particular positions and what percentage of bases are covered at a particular depth at each of those positions to really understand the performance of your assay. But thinking more about our particular regulatory requirements from CAP and CLIA, there are specific performance characteristics that must be assessed for all assays and in particular for NGS assays. These include accuracy and precision, analytical sensitivity and specificity, as well as the reportable range and the reference range. If you were using a kit from a vendor and happen to have an FDA-approved or cleared kit, it may be less important to assess all of these particular characteristics. However, they should be verified and the performance of the assay should be confirmed before launching clinically. When we think about the particular types of variants we want to assess, we want to make sure that we validate every one of the variant types and we validate each of the characteristics. And this can be a little bit complicated when we think about what type of matrix of samples we want to create, because really we have five different variant types potentially and we have six different characteristics. Fortunately, many of our samples harbor many different types of variants and we are actually assessing all of these characteristics simultaneously in each run. Going back to the Jennings paper, which talks about development of a validation for an oncology panel, it's important to think about designing a sample matrix for performing of your validation. So you set it up with multiple sequencing runs, you have different technologies for your between tech reproducibility studies, and then in this example there are 12 different samples that can be loaded per run, and there's a variety of previously tested patient samples and limited detection samples. And by performing these runs after thinking about the validation matrix, we're able to optimize how we do our clinical validation. And one of the big questions we often face when we're trying to do validation is where do we get the best samples? So patient samples are considered the gold standard, however they require orthogonal confirmation from a validated assay, and fortunately in many ways we can use samples that were previously sent to a reference lab and compare the results that we get to those from the In addition, the use of cell lines and synthetic multiplex controls are available from a variety of vendors, some of whom are shown here. It's also possible to use in silicose samples, as I mentioned earlier when we're looking at the validation of our informatics pipelines. But it's important to understand that when we get our samples, we want to have a mixture of positive and negative samples. We want to know what the variant allele fraction is in the sample that we're testing, so we have a sense of how our assay performs, and we want to make sure that we cover all the variant types that are clinically relevant in our assay. In principle, how does this work? Here we have some previously tested samples, and let's say 1 to 40 are positive, where 1 to 20 have SNVs, and 21 to 40 have single or multiple indels. 41 to 60 are negative samples, or no variant samples, and then LOD 1 through 3 are multiplex controls with all the different variant types, and we know what the performance is. So if we want to look at the accuracy for single nucleotide variants here, we would look at the positive control samples that have single or multiple single nucleotide variants. So in this set of 20 samples, we have 35 SNVs total. Our assay accurately detects 34 of these, and in the negative samples, our assay correctly calls 18 of 20 negative, with 2 false positive SNVs detected. In our multiplex control samples that are also being tested, we have 18 SNVs tested, and our assay accurately calls all 18. So putting this into our standard table for assay performance, we see that we have a positive percentage agreement of 98%, and a positive predictive value of 96%. So this is relatively good performance of this assay, although this is a somewhat limited assessment, as we're only looking at SNVs, and as I mentioned, this particular assay, we want to look at complex variants, and so we need to look at those samples 21 to 40 as well. Now let's calculate the precision for the SNVs. Now let's calculate the precision for single nucleotide variants in our particular assay. So when we look at the repeatability of the testing, we think about within-run precision, so we want to look at the performance of samples that are in yellow or in green. So these are the samples that are repeated within a run, here in yellow and in green. When we think about the reproducibility, we want to think about the between-run precision, and here we have samples that are in blue or in green. So here we have the PS1s are repeated across three different runs, here the PS2s are repeated across three different runs, etc. And when we calculate the precision, we want to think about making sure that the variant allele fractions are somewhat similar, and the depth of coverage is similar, and we want to calculate our mean and standard deviation for our reproducibility of these variants. Next, we want to think about the analytical sensitivity and specificity for SNVs in this particular assay, and we want to think about the limit of detection and what the input limit testing is. And in particular, we want to think for specificity about particular interferences that may impact this. If we're thinking about formalin-fixed paraffin-embedded tissues, we may want to think about the presence of melanin pigment or alternate fixation that may lead to impaired performance of the extracted nucleic acid. And here we can see that we have an assay that has a very good performance for the limit of detection, where down to a very low variant allele fraction, the observed variant frequency tightly mirrors the expected variant frequency. Next, when we think about defining the reference and the reportable range, it's important to think about both what the reference genome is that we're using, and there's a lot of normal variation. Here we have information from the HG38 genome, and we see that this is normal variation, or changes that we're seeing in the HG38 update to the reference genome, and so whatever is present in the reference genome as wild type or normal is considered our reference range for nucleic acid sequencing. However, it's also important to understand the presence of normal variation, and so to this we turn to population databases such as the Genome Aggregation Database, or NOMAD, as well as many others. When we think about the reportable range, we want to think about really the regions with the acceptable assay performance. So as I mentioned earlier, there may be regions of the genome that we were trying to target by the assay, but they're difficult to sequence, and we recognize during the validation process that these areas cannot be accurately sequenced. We may decide to shrink the reportable range for the assay based on the inability to obtain acceptable performance at those regions, or we may shrink it based on the absence of any clinically meaningful variants in those regions. All of this should be included within the method description and disclaimer in the clinical report. And then something that we've started to think about a lot more in recent years is clinical sensitivity versus analytical sensitivity, and clinical specificity, and analytical as well. So when we think about the analytical performance, the analytical sensitivity is the ability to detect a mutation or disease, but when we think about it clinically, it's the likelihood that a positive result correctly determines that the patient has the condition being tested. So this is going to be impacted by things like the size of the panel and the ability of your assay to detect all of the potential variants that are associated with a particular disease. This is simple in monogenic diseases like cystic fibrosis, but it can be complex in multigenic disease. We think about analytical sensitivity, we're thinking about the ability of a normal or negative result to correlate to specimens that don't harbor a mutation or disease. But clinically, we're thinking about it as the likelihood that a negative test result correctly determines that the patient is healthy or does not have the condition. Again, this is going to reflect how much of the actual genetic disease variability is being assessed by the particular assay. Analytical sensitivity and specificity can be determined by using validation data. However, it's important to use many databases and the literature to help assess the clinical sensitivity and specificity of a particular test. As we're starting to run out of time, we will briefly talk about the ideas of quality management and ongoing assessment for the clinical assay. To ensure that an assay performs robustly, it's imperative to continue to do routine quality control on every run, and this includes the incorporation of control samples with the run, review of the sequencing metrics generated by the sequencing instrument, as well as review of the bioinformatics methods being generated by the pipeline. As well as the routine QC, it's important to enroll in proficiency testing to ensure that the laboratory is able to accurately detect variants in unknown samples. And as I mentioned earlier, we are starting to incorporate ideas of in silico validation or in silico samples for our validation and the use of methods-based testing for our samples. These principles have also been incorporated into proficiency testing rubrics. In summary, next-generation sequencing assays are complex. They include considerations of the particular sample type and its limitations, which library prep is going to be most effective for a particular clinical implementation, how to sequence and analyze the performance of sequencing, and then the ongoing data analysis, both from automated informatics pipelines all the way through human annotation and analysis for reporting. Clinical requirements that we develop for our assay inform the design requirements, and this includes the particular targets or the regions of the genome that need to be sequenced to detect variants associated with a disease, understanding what particular variant types we need to detect, whether it's only simple variants like single nucleotide variants or if we need to think about copy number variants, rearrangements, fusions, and large indels. And lastly, the sensitivity that's clinically informative is important because it informs the depth of sequencing and the ability to multiplex our samples. And as we move our samples from our assays from design to validation, it's important to consider including an optimization and familiarization phase to improve the performance of the final assay. This is sort of a mini-validation, which allows us to understand the particular performance characteristics and limitations of our assay, and because it's before the final validation, we're able to use this time to optimize and improve particular areas of performance, either by changing the number of samples being incorporated in a run or by potentially altering some of the particulars of the wet bench process. And lastly, validation defines our final assay performance, and this defines specific classifications such as the accuracy, precision, analytical, and clinical performance, as well as the reportable and reference ranges, and this is critical both from a regulatory and legal standpoint but also to ensure that we are generating good data for all of our patients. At this point, I'd like to thank you for your attention and have a nice day. So with that, I'd like to thank you for your attention and open the floor for questions.

NGS assays

test development

test design

test validation

quality management

sample types

sensitivity

sample throughput

turnaround time