Clinical Pharmacology Quality Assurance (CPQA) Program: Models for Longitudinal Analysis of Antiretroviral (ARV) Proficiency Testing for International Laboratories (2024)

Abstract

Keywords: longitudinal analysis model, antiretroviral proficiency testing, inter-laboratory performance

INTRODUCTION

The quality of clinical pharmacology in HIV research is of global concern. The National Institutes of Health support multiple HIV clinical research networks and investigator-initiated clinical trials to sponsor the investigation of treatment paradigms in diverse international populations of HIV+ and HIV− individuals. Most of these investigations involve the use of drug treatment for infected patients as well as the use of drugs for the prevention of infection in those at risk. The pharmacology objectives of these research studies require extensive bioanalysis for drug concentrations. Furthermore, cross-protocol analyses frequently employ these drug concentrations to address additional questions about pharmaco*kinetics, pharmacodynamics and pharmacogenetics. Because these studies impact the strategies used for treatment and prevention, the quality of the drug measurements is critical.

Through the Clinical Laboratory Improvement Act (CLIA) of 1988, the US federal government mandated that clinical laboratories participate in and demonstrate satisfactory performance in proficiency testing (PT) programs [1]. However, for clinical pharmacology laboratories, unless the drug concentration is used clinically to treat the patient, the assay of drug concentrations in plasma and other matrices performed for drug development and research purposes falls outside the CLIA mandate. The European proficiency testing program International Inter laboratory Quality Control Program for Measurement of Antiretroviral Drugs in Plasma, has been conducted for more than a decade. The program was established by Radboud University Nijmegen Medical Center, The Netherlands, and later continued in collaboration with the Dutch Association for Quality Assessment in Therapeutic Drug Monitoring and Clinical Toxicology. This program was needed in Europe as therapeutic drug monitoring (TDM) for antiretrovirals (ARV) was a standard in clinical treatment for HIV+ patients receiving ARV [2, 3]. The European program analyzed laboratory PT results and published results in 2002, 2003 and 2010¹[4–6]. The most recent report was for ten years of PT results, exhibiting 84% accuracy, and somewhat lower accuracy(78%) in samples at lower prepared concentrations. In addition, accuracy varied by individual ARV. A multivariate model of these two factors was not fit; however, it was apparent that those ARVs with higher therapeutic concentration ranges (efavirenz, lopinavir) were more likely to have greater accuracy than those with lower therapeutic concentration ranges (saquinavir, atazanavir). A similar program has been implemented by this organization for antifungals [7].

A US-based ARV PT program, operated by the ACTG/IMPAACT from 2000–2008, has also published PT program results [8, 9]. This program was a collaboration of multiple university CPLs analyzing ARV concentrations in pediatric and adult clinical trials. Two of the participating universities prepared and shipped PT samples with one of the two coordinating the compilation and distribution of reports. The program maintained an overall accuracy of 96%.The program also concluded that the measure of accuracy was dependent on the ARV and laboratory. Initial data indicated that the concentration of the ARV, classified as “low”“medium” or “high” spike per ARV, was a significant variable. However after additional PT results were obtained, concentration was no longer found to be a significant variable. This program was discontinued in 2008.Table 1 summarizes some key characteristics, methods and performance results reported in the literature for the US and European ARV PT programs. Many fundamental differences exist between the two programs, such as number of samples per PT event (herein referred to as “round”)and the statistical methods employed in the analyses.

The Clinical Pharmacology Quality Assurance and Quality Control (CPQA) program was established in 2008 and provides multiple quality-centered activities for all NIAIDHIV Clinical Trial Networks globally. Quality strategies include: (1)training curricula for clinical sites conducting studies and for laboratories analyzing specimens; (2)confidential, anonymous, peer-review of assay validation reports submitted by participating CPLs for approval to assay clinical trial pharmacology samples ;(3)a phased laboratory assessment system, and; (4)a PT program for ARV measurements. This program was established in May 2008 and is currently ongoing. The CPQA PT program includes multiple CPLs that report ARV concentrations; for each ARV 2 to 10 CPLs reported results during the first 2 years of testing. Although all CPLs use liquid chromatography, the reagents, calibrators, equipment, consumables and detection method differ. Recognizing the lack of statistical models used to quantify variances in laboratory PT programs such as this, CPQA developed new models testing differences in recovery among CPLs and analytes, to contrast two alternate ways of assigning target value and to compare regulatory and model-based statistical limits of acceptance.

Most often, the drug quantitation tests to which the CLIA PT mandate applies include “high volume” tests such as phenytoin and digoxin. For such tests, the number of participating labs is large enough that PT performance assessments are applied separately to different “peer groups,” that is, labs that use common (or very similar) measurement procedures [1, 10]. In simulation studies, researchers have evaluated both the original CLIA rules for PT evaluation and newly proposed rules for diagnostics that individual labs might apply to their own PT data to assess and track systematic and random error in their tests. Such rules are defined by (1) the assignment of target values, (2) the acceptance criterion used (eg, regulatory, statistical or clinical) and (3) how scores for the 5 individual PT samples are reduced to the single test result of “satisfactory” or “unsatisfactory”. These simulation studies have yielded important findings regarding which rules work best in situations of high- and low-noise tests; however, these studies have assumed a large enough number of participating labs that group means, medians and standard deviations are not affected by a single lab whose measurement procedure exhibits high systematic or random error [1, 11, 12].

The CPQA primary goal for PT is to assure the accuracy of CPL reported concentrations (RCs); PT reports flag deviations from target concentrations in the current round and assign each CPL a score based on performance in the current and 2 prior rounds. However, goals of the program and the performance of the CPLs may be enhanced by a comprehensive, longitudinal analysis of ARV PT data, the analysis of which may suggest ways to improve the policies and practices of both laboratories and the PT program. The objectives of this longitudinal analysis of PT results across multiple CPLs and rounds were to develop and test statistical models, applied to longitudinal data, in order to (1)assess the precision and accuracy of concentrations reported by individual CPLs; and (2) determine factors associated with round-specific and long-term assay accuracy, precision and bias. In addition, the metric of absolute recovery was explored as a simultaneous measure of accuracy and precision. The outcomes achieved by the proposed models were interpreted for PT relevancy and impact.

METHODS

Every six months the CPQA PT program offered prepared plasma samples containing pre-specified concentrations (unknown to CPLs) of up to 21 ARV analytes: abacavir (ABC), amprenavir (APV), atazanavir (ATV), darunavir (DRV), didanosine (DDI), efavirenz (EFV), emtricitabine (FTC), etravirine (ETR), indinavir (IDV), lamivudine (3TC), lopinavir (LPV), maraviroc (MVC), nelfinavir (NFV), nevirapine (NVP), raltegravir (RGV), ritonavir (RTV), saquinavir (SQV), stavudine (D4T), tenofovir (TFV), tipranavir (TPV), zidovudine (ZDV). In each round and for each ARV, 5 concentrations, spanning an expected therapeutic range of each ARV, as well as occasional concentrations below or above, were provided. Samples are prepared by an outside subcontractor and tested by the CPQA lab prior to distribution. PT samples were stored at −70 ± 15°C and then shipped on dry ice to participating laboratories with detailed instructions. Upon arrival, each laboratory confirmed sample integrity and indicated planned reporting of specific analytes. Results were reported either through an online Laboratory Data Management System (LDMS) or via a template which was then uploaded into the LDMS database. At the end of the submission period, a completeness evaluation was performed to confirm that all planned results were received; discrepancies were queried for resolution. To summarize the proficiency of individual labs, a pre-specified scoring algorithm was applied to the RCs (see next paragraph). The scoring algorithm reflects US Clinical Laboratory Improvement Act (CLIA) PT regulations[13]. After review and approval by the CPQA advisory board chair, a final report was sent to the participating laboratories (with laboratories de-identified) and key leadership (laboratories identified per network leader).

An individual RC is deemed Acceptable provided a concentration is present where expected, and the concentration is within 20% of the final target (FT)[14]. (If a concentration is reported as below the lower limit of quantification (BLQ), and the run lower limit was below 80%*FT, the concentration was labeled Unacceptable.) For a given prepared sample, if the number of labs reporting for that sample is large enough, the variability between CPLs is small enough (≤15%) and the percent deviation of the group mean (GM, determined after removal of outliers, if any)from the weighed-in value (WIV) is >5%, the FT is set to the GM. Otherwise, FT is set to the WIV. At the analyte level, a CPL’sperformance is deemed Satisfactory for the round provided at least 80% of RCs are Acceptable. If the CPL score is <80% for an analyte, the CPL submits a corrective action plan to reestablish accuracy; a root cause is requested. Finally, in accordance with CLIA rules, a lab is classified as successful for an analyte provided the round-specific score was Satisfactory in at least 2 of the last 3 rounds (including the current).

RESULTS

Table 2 summarizes the results of the algorithm used to determine FTs. For 47% of unique samples, the WIV was chosen as the FT. The percent of samples with FT=WIV varied across analytes, and ranged from 0 to 100%. For analytes 3TC, DDI, DRV, NVP, RGV and TPV, the WIV was used for 20% or fewer samples; for all other analytes, the WIV was used for at least 40% of samples. In the case of DDI, the selection of GM as the FT was driven by a small number of RCs (DDI was offered in only one round). In the case of the more frequently reported DRV and NVP, choice of GM was driven by large deviations between RCs and the respective WIVs.

Table 3 enumerates the number of results per analyte per CPL and also provides totals by analyte and CPL. Ten laboratories participated in the PT program, and reported results for 6 to 20 ARV. Six CPLs participated in all four rounds. CPLs 1, 3, 8 and 9 began participation at later dates. CPL participation per ARV ranged from 20% (2 CPLs reporting for ETR) to 100%(many CPLs). ATV, DRV, EFV, FTC, LPV, NFV, NVP, RTV, and TFV were reported more frequently than the other ARV (with ≥90 results reported and at least 7 CPLs reporting for each ARV). For each drug, 95% of RCs were Acceptable except for FTC (90%), NFV (89%) and TFV (92%). Pooling RCs over all rounds, 97% were Acceptable. All RCs for CPLs 5, 8 and 10 were Acceptable; for CPLs 3 and 6, 91% and 94% of RCs were Acceptable; and for the remaining CPLs, at least 97% of RCs were Acceptable.

Satisfactory scores were achieved for 97% of CPL analytes/rounds. All scores for CPLs 1, 5, 8 and 10 were Satisfactory; for CPL 3, 91% of scores were Satisfactory; and for the remaining CPLs, at least 95% of scores were Satisfactory. Similarly to RCs, Satisfactory scores of <95% were seen for the analytes FTC (86%), NFV (83%) and TFV (83%).

Six of the ten CPLs failed to meet criteria for one or more analyte in one or more rounds (not tabulated). These CPLs submitted nine analyte-specific corrective action plans for failure to meet proficiency standards during a round. Root cause analyses by CPLs indicated that three main sources of error occurred: technical/human error (40%), assay calibration problem (33%), and failure to follow assay standard operating procedure (27%). Laboratories that participated in 3 or more rounds per analyte maintained CLIA proficiency².

Of the 1,706 results, ten were omitted from the statistical analyses due to errors unrelated to assay performance. These errors included clerical errors such as incorrect or reversed transcription and extrapolation over the calibration range limit. Thus, 1,696 concentrations were included in analyses.

DISCUSSION

The overarching aim of the PT program is to provide a means by which the stakeholders in HIV research (funding agency, investigators, patients, and health care providers) can be assured that pharmacology assays conducted for clinical trials are sufficiently accurate and reproducible so that minimal bias and minimal variability are introduced into ultimate findings and conclusions of these clinical trials. Currently, a twice-yearly PT program addresses overall and analyte-specific performance of individual labs conducting pharmacology assays, providing warnings for each round as needed if performance is unsatisfactory. Performance over three rounds is used to determine CLIA status and serves as a longitudinal view.

Without employing some type of statistical analysis, the general bias over four rounds, or two years, is not evident. In models examining the potential for bias, recovery relative to the nominal concentration, on the log scale, gave values that were approximately normally distributed. A linear regression model fit to these data identified an overall tendency of slight overestimation of concentrations (positive intercept estimate) and a global tendency for larger overestimation at larger nominal concentrations (positive slope estimate). Using a prediction interval around the estimated regression line provided an alternate to the current acceptance criteria. By this procedure an additional 54 (3.2% of 1706 reported concentrations) would be considered unacceptable, and 8 concentrations considered unacceptable by the current criteria would be deemed acceptable, changing the overall rate of acceptable concentrations from 97.6% to 94.9%. Using the group mean (mean over labs) as the nominal concentration (for about half of the prepared samples, as was done by the PT program) attenuated estimates of bias overall and as a function of concentration magnitude. Prediction intervals from the regression of recovery on final targets (i.e., when the group mean is used when use of the WIV is not indicated) would classify 95.6% of concentrations as acceptable (compared to 97.6% from the existing PT program criteria). These results suggest that a smaller window of acceptance should be considered. Continuous monitoring the PT data longitudinally and/or simulation studies will assist the CPQA in making this decision.

Pooling results over four rounds (but not yet applying statistical models), it appeared that CPL 3 and 6 had more difficulties than the other CPLs, as evidenced by lower rates of acceptable reported concentrations and of satisfactory scores (≤95%). Using regression models to examine associations of CPLs identified with bias indicated higher recoveries for CPL 6 and 7 (but not 3) compared to the other labs and CPL 2 was identified as having lower recoveries as compared to other CPLs. This finding held regardless of which nominal concentration was used (i.e., held for both models 2A and 2B). CPLs 3, 6 and 7 (but not CPL2) were also deemed to exhibit higher error (as measured by absolute recovery) compared to the other labs, regardless of which NC was used (models 3A and 3B).Pooling and tabulating results of the four rounds, ARVs TFV, FTC, and NFV appeared to be most challenging to assay, as evidenced by lower rates of acceptable reported concentrations and of satisfactory scores (<95%). However, using regression models to examine associations of analyte with bias, reported concentration for DRV and RGV were significantly higher than WIVs but not the final targets. Since recoveries using weighed-in value (Model 3A) were high, (ranks of) absolute recovery (errors) were noted to be largest for these ARVs as well. And although the final target was adjusted to the group mean for 90% of DRV and RGV specimens, the adjustment did not correct for error and DRV and RGV remained high in the ranks (1 and 3, respectively) of absolute error. NVP was identified as second in the rankings for model 3B inferring that in the absence of bias, error can still be large. If the ARV exhibits both positive and negative deviations from the final target, bias tends to cancel out, but persist in absolute recovery, indicating variability. In the presence of bias, errors can also be large when using WIV or FT in which case the contribution of variability to error is difficult to tease out.

A discussion of the differences in findings by analysis approach is warranted. All three methods (pooling and tabulating, and regression-based assessments of bias in recovery and error (regardless of which NC was used) identified CPLs 3, 6 and 7 as having somewhat poorer performance than the other labs. However, conclusions about analytes differed between methods. Pooling and tabulating over the four rounds pointed to problems with the assay of FTC, NFV and TFV, whereas models of bias (recovery deviating from 100%) indicated problems with the assay of DRV and RGV when WIV was used as the NC, and no problems when FT was used as the NC. Models of error (ranks of absolute recovery) also pointed to problems with DRV and RGV assay when weighed-in value was used as the nominal concentration, and problems with DRV and NVP when FT was used at the NC. The large discrepancy between reported DRV concentrations and weighed-in values could reflect either preparation error or an error in assay methodology that is common across some but not all labs. While errors common across labs may seem less feasible, it was known that all labs use the same source for reference powders and prepare calibration standards in a similar fashion. Near the last of the four rounds included here, multiple laboratories (using ultraviolet detection) have reported secondary peaks emerging from their standard calibration stocks as a potential source of error; this was reported on CPQA-held cross-network conference calls of network CPL supervisors held every other month. For 90% of DRV samples, the group mean was used as the target, which “protected” labs from failing. In the case of preparation error, this protection is desirable. For the DRV results included here, this correction may be considered inappropriate. Hence, the choice of nominal concentration can be problematic. In contrast, although reported concentrations for FTC were near the WIV (unbiased), some labs did fail for the ARV. However, agreement among the majority of labs supported the accuracy of sample preparation and laboratory methodologies.

Other factors examined were round and 2 categorizations of ARV concentration. In most cases, these variables did not exhibit significant associations with bias and/or error, except for Model 2A which showed significant association with concentration (both categorizations) and Model 3A which showed significant association with round. In the latter case, the source or error most likely from arises from preparation of PT samples, where weighing for solutions to prepare PT samples are new from round to round.

While the regression models employed here may be valuable tools to diagnose potentially problematic assay factors, results from the model depend on which quantity is used as the nominal concentration. Use of the WIV as the nominal concentration has the desirable statistical property that it is independent of the dependent variable “reported concentration.” However, in the event of preparation error, the WIV itself is biased from the true but unknown concentrations in the PT samples. When the number of CPLs reporting for an analyte is large enough (4 or more), when the variability among labs in reported values is small (≤15%) and when the deviation of the group mean (mean reported concentration across labs) is large (>5%), the group mean is used as the target concentration – the determination of acceptable/unacceptable is relative to this group mean, not to the WIV concentration. Substituting a data-driven target for the prepared concentration, in these instances, is designed to prevent application of the “unacceptable” label when there was preparation error. However, a drawback in using this final target as the so-called independent variable is that, when the group mean is substituted, the independent variable is not statistically independent from but is based on (subsets of) reported concentrations (for some samples), the dependent variable. Furthermore, this substitution could give an advantage to some labs. For example, given the discussion of DRV PT above, if multiple labs used the problematic reference powder and a single lab used a no problematic reference powder, the single lab could be penalized by this choice of nominal concentration. Future research will investigate the use of the median reported concentration as the independent variable; this is the approach taken by the DAIDS Virology Quality Assurance program [15].

The data at hand are unbalanced, in the sense that labs participated for different subsets of analytes and in a different numbers of rounds. If recovery truly varies by analyte, then differential participation by labs (some labs reporting for “problematic” analytes and others not) would tend to unfairly penalize the former type of lab when using a statistical model that does not simultaneously estimate the effects of analyte and lab. In future research, we will examine models that account for multiple factors (e.g., both analyte and lab) simultaneously. These models will need to use appropriate estimation and inference techniques in the situation of unbalanced data.

CONCLUSIONS

Given that a multiplicity of variables both within and outside of laboratory control can affect results, proficiency testing in the context of drug assays for HIV clinical trials is not a pure measure of laboratory-specific accuracy or reproducibility. Application of statistical models to multiple rounds of proficiency testing data sheds light on the potentially inaccurate conclusions made by observing only “Acceptable” or “not Acceptable,” and focuses attention on bias and error as sources of problematic laboratory (or laboratory method) performance. Using two different measures of nominal concentration (WIV or FT as determined by a data driven algorithm), the complexity of identifying the relationships between bias and error, as well as laboratory and ARV, is apparent.

With respect to the current information these models have provided, laboratory and analyte remain the most significant variables whether they are associated or not. It is noteworthy that the CPLs maintained CLIA proficiency for ARV quantitation with excellent scores. This approach to longitudinal analysis of proficiency testing provides novel insights into inter-laboratory characteristics that are key components to the crucial role of the CPLs in improving the quality of assay performance and for using data from different studies to accomplish the clinical pharmacology research goals of the NIAID HIV research networks.

ACKNOWLEDGEMENTS

Footnotes

REFERENCES