19. Closeout

Many details of closeout will depend on factors that only become known once the trial is underway or participant enrollment is completed. Nevertheless, general planning for the closeout ought to start early. There are arguments for initiating this process on day one of the trial. Data management processes can be optimized for rapid database finalization at the end the trial. One major issue is that the trial may not continue through its scheduled termination. Greater-than-expected benefit or unexpected harm may lead to early termination. A more subtle reason is that developing plans for closeout after the trial is well underway may be interpreted by the blinded investigators as a signal of imminent trial termination. Thus, another recommendation is to develop the general closeout plans prior to the first meeting of the data monitoring committee [1].

The closeout phase needs its own written protocol or operating procedures with respect to termination activities, dissemination of results, and data cleanup and storage. The literature on the topic of closeout is scant but there are a few good descriptions of the process [2].

If each participant in a clinical trial is to be followed for a fixed period of time, the closeout phase will be of the same duration as the enrollment phase. If recruitment took 2 years, the closeout phase would last 2 years. This fixed follow-up design may not be desirable, since terminating the follow-up of some participants while others are still being actively followed can create problems. In some blinded trials, the code for each participant is broken at the last scheduled follow-up visit. If the unblinding must occur over a span of many months or years, there is the possibility of the investigator learning information that could suggest the identity of the drugs taken by participants still actively followed in the trial. This may happen even if drug codes are unique for each participant. The investigator may start associating a certain symptom or constellation of symptoms and signs with particular drug codes.

An alternative and frequently used plan involves following all participants to a shortened closeout period to avoid the problems described above. Another advantage of this design is the added power of the trial and more information about the effects of longer intervention. The follow-up period is extended beyond the minimum time for all but the last participant enrolled. In a trial with 2 years of uniform recruitment, the additional follow-up period would increase by an average of up to 1 year. In addition, this approach might be more cost-efficient when clinic staff is supported solely by the sponsor of the trial. With all participants followed to a shortened closeout period, full support of personnel can be justified until all participants have been seen for the last time. In trials where the participants are phased out after a fixed time of follow-up, an increase in the staff/participant ratio may be unavoidable.

Despite the problems with following all participants for a fixed length of time, this approach may be preferable in certain trials, particularly those with a relatively short follow-up phase and when the effect of the intervention is believed to be restricted to a short period of time. In such studies, there may be no realistic alternative. In addition, it may not be logistically feasible to conduct a large number of closeout visits in a short time. Depending on the extent of data collection at the last visit and availability of staff and weekly clinic hours, seeing 100–150 participants at a clinic may require a month or 2. A decision on the type of follow-up plan should be based on the scientific question as well as logistics.

At trial termination, it is important in any trial to obtain, to the extent possible, response variable data on every enrolled participant. It is particularly so in trials where the main response variables are continuous, such as laboratory data or a performance measure. By necessity, the response variable data must be obtained for each participant at the last follow-up visit because it marks the end of treatment and follow-up. If the participant fails to show up for the last visit, the investigator will have missing data. When the response variable is the occurrence of a specific event, such as a nonfatal stroke or death, the situation may be different if the information can be obtained without having the participant complete a visit.

Another approach is to have a common cut-off date (for example, the date of the first planned final follow-up visit). A common cut-off date—such as was used in the Apixaban for Reduction in Stroke and Other Thromboembolic Events in Atrial Fibrillation (ARISTOTLE) trial [3]—may be an advantage for clear definitions in the statistical analysis and for dealing with participants who may never appear for their final visits. In favor of a variable cut-off date (used in most trials) is that including all events until the last participant’s final visit allows maximal capture of the exposure period and therefore optimizes the precision of the estimate of the intervention effect. It is possible that the timing of the final visit could be affected by assignment to intervention or control and therefore could create bias when a variable cut-off date is used, although this would likely be small.

A number of means have been used to track participants and to determine their vital status. These include the use of a person’s identification number (e.g., Social Security number in the United States) or contact with relatives, employers, or health care providers. It has been discovered that participants have died through searching obituaries. Electronic medical records and various other electronic databases can be searched, with appropriate permissions. In countries with national death registries, including the United States, mortality surveillance is simpler and probably more complete than in countries without such registries. Agencies that specialize in locating people have been used in several trials. As has been used in many trials, the Digitalis Investigation Group trial [4] used a search agency, but the searches were limited to records only. It used directory assistance, credit header reports, property records, obituary searches, database mailing lists for magazine subscriptions, and other similar means. No personal contact was allowed. These constraints probably limited the success of finding participants lost to follow-up. This process can be sensitive, since a search may be looked upon as an intrusion into the privacy of the participant. The integrity of a trial and the importance of its results plus the participant’s initial agreement to participate in the trial have to be weighed against a person’s right to protect his or her privacy. Investigators should consider including in the informed consent form a sentence stating that the participant agrees to have her vital status determined at the end of the trial even if he or she has by then stopped participating actively or withdrawn general consent. It pays to initiate the process of obtaining information on vital status on inactive participants well in advance of the closeout phase.

The uncertainty of the overall results rises as the number of participants for whom response variable data are missing at trial termination increases. For example, assume that death from any cause is the primary response variable in a trial and the observed mortality is 15% in one group and 10% in the other group. Depending on study size, this group difference might be statistically significant. However, if 10% of the participants in each group were lost to follow-up, the observed outcome of the trial may be in question.

The Anti-Xa Therapy to Lower Cardiovascular Events in Addition to Standard Therapy in Subjects with Acute Coronary Syndrome–Thrombolysis in Myocardial Infarction 51 (ATLAS ACS 2–TIMI 51) trial of rivaroxaban following acute coronary syndromes [5] highlights the importance of completeness of follow-up data. The trial had a primary outcome of cardiovascular death, myocardial infarction, and stroke, using an on-treatment plus 30-day analysis with an intention-to-treat sensitivity analysis. In the original data from which the primary manuscript was published and the first United States Food and Drug Administration (FDA) filing was made [5], 1509 (or about 10%) of 15,526 participants had incomplete follow-up, and 799 participants had incomplete follow-up limiting the observation period of up to 30 days after early discontinuation. For the primary analysis, there was a 1.8% absolute and a 16% relative risk reduction with rivaroxaban (p = 0.008) that was counterbalanced by more major bleeding. There was a 0.8% lower mortality with rivaroxaban than placebo (p = 0.04) in the primary analysis. The FDA review raised important issues about missing data [6]. First, when the FDA declared that the 10% missing data made it impossible to interpret the mortality data, the sponsor was able to go back to the sites and establish vital status for 843 of 1338 patients with a missing status at the end of the trial, showing that it was possible to have more complete follow-up with more intense effort. When this was done, 22 additional rivaroxaban and 9 additional placebo participants were found to have died, and the p-value increased from 0.045 to 0.076 (for the main “stratum 2” with background thienopyridine therapy). Secondly, it was more common to have missing data in the rivaroxaban arm than in the placebo arm, raising further questions about interpretability. Third, there were three sites in India with questionable data that could neither be verified nor proven to be fraud, which raised additional questions about whether to exclude these data. To address the potential impact of the missing data with regard to mortality, the FDA presented a variety of imputation scenarios to address the missing data in stratum 2 [7] (see Table 19.1). Using the conservative approach of assuming the same mortality for all patients with missing data, the p-values increased from 0.076 to 0.100.

Another example of the problem of participants being lost to follow-up, and specifically of withdrawal of consent, comes from the Comparison of Medical Therapy, Pacing, and Defibrillation in Heart Failure (COMPANION) trial [8] of defibrillator versus pacemaker versus best medical care. Withdrawal of consent was four times higher in the medical care group than in the other two groups when the trial was terminated and follow-up ended. At a recommendation by the data safety and monitoring board, the investigators approached the participants who had withdrawn their consent and obtained their permission to collect data on vital status and hospitalizations retrospectively for the duration of the trial. This was done at a substantial extra cost and loss of time and stresses the importance of prevention of withdrawal of consent.

It is a mistaken concept that when a participant goes off study medication or intervention that he or she is out of the study and thus no longer followed, or at least not followed beyond some short period of time such as 7 days and 30 days. In the Adenomatous Polyp Prevention on Vioxx (APPROVe) study, participants who stopped their study medication (rofecoxib) due to adverse effects and other reasons were not followed beyond 14 days of going off drug [9]. In the re-analysis, the problem with this “informative censoring” was revealed and a full extra year of follow-up of all randomized participants after stopping study treatment was added. This analysis suggested that the excess of drug-induced major cardiovascular events observed during rofecoxib treatment continued to increase during the first year after treatment was stopped. The adjusted hazard ratio for the extra year was 1.41 (95% confidence interval [CI] 0.77–2.59), in addition to the hazard ratio of 2.12 (95% CI 1.20–3.74) on treatment and during the following 14 days.

The National Academy of Sciences, prompted by FDA officials, has published a comprehensive statement concerning missing data in clinical trials with a focus on phase III confirmatory trials [10]. The report states that “there is no ‘foolproof’ way to analyze data subject to substantial amounts of missing data; that is, no method recovers the robustness and unbiasedness of estimates derived from randomized allocation of treatments. Hence, the panel’s first set of recommendations emphasizes the role of design and trial conduct to limit the amount and impact of missing data” [10]. We stress the need to have systems in place from the beginning of the trial to minimize missing response variables, which for phase III trials begins with carefully structuring the informed consent to allow follow-up, at least for vital status, even if a participant otherwise withdraws from study procedures.

Verification of data may be time-consuming, and it can conflict with the desire of the investigator to publish his findings as early as possible. While publication of important information should not be delayed unnecessarily, results should not be put into print before key data have been verified. Despite attempts to collect complete, consistent, and error-free data, perfection is unlikely to be achieved. Traditional monitoring systems are likely to reveal missing forms, unanswered items on forms, and conflicting data. In isolated cases, they may also uncover falsification of individual data [11, 12] and, in the worst cases, fabrication of all data on fictitious participants [13–15]. Data cleanup and verification typically continue for months after completion of the closeout visits, although the use of electronic records has substantially reduced the burden of this cleanup and verification. It is necessary to be realistic in the cleanup process. This means “freezing” and “locking” the files at a reasonable time after the termination of participant follow-up and accepting some incomplete data. Obviously, the efforts during cleanup should be directed toward the most critical areas—those crucial to answering the primary questions and serious adverse effects.

We strongly recommend that study forms and data be continuously monitored throughout a trial as pointed out in Chap. 11. Data editing should be initiated as soon as possible, because it is difficult to get full staff cooperation after a trial and its funding are over. Early monitoring may reveal systematic problems that can be corrected. Staff feedback is also important. Approaches for statistical process control audits are now available, and they have been shown to reduce the overall database error rates significantly [16].

Any clinical trial may be faced with having its results reviewed, questioned, and even audited. Traditionally, this review has been a scientific one. However, since regulatory and special interest groups may want to look at the data, the key results should be properly verified, documented, and filed in an easily retrievable manner. The extent of this additional documentation of important data will depend on the design of each trial. Electronic data provide verification opportunities that are more efficient than paper records, but storage remains important. Various models have been used. In one multicenter study, the investigators were asked at the end of follow-up to send a list of all deceased participants along with date of death to an office independent of the data coordinating center. In other trials, key data were independently audited before the results were published. Common to all models is an attempt to maintain credibility.

Procedures for data cleanup and verification in trials conducted for regulatory approval add substantially to the trial cost and complexity. Many such trials collect a large quantity of data. Final verification of these data is both time-consuming and costly [17, 18]. As noted in Chap. 11, investigators should, when designing such trials, both limit the amount of data and decide which data are essential and require full final verification.

Investigators should consider storing various kinds of material after a trial has ended. One set of documents—such as the trial protocol, manual of procedures, study forms, and the analytic material, including electronic records—should be kept by the investigator and sponsor. In addition, a list containing identifying information for all participants who enrolled in a trial ought to be stored at the institution where the investigation took place. Local regulations sometimes require that individual participant data such as copies of study forms, laboratory reports, electrocardiograms, and X-rays be filed for a defined period of time with the participants’ medical records. Storage of these data electronically clearly eases the problem of inadequate space. The actual trial results and their interpretation should be published and then can be retrieved through a library search, although it is all-to-common for trial results to remain unpublished [19]. As of 2012, less than two-thirds of National Heart, Lung, and Blood Institute (NHLBI)-sponsored clinical trials were published within 30 months [20]. Recognition of this major problem, pressure from sponsors and the clinical trial community to publish all trial results, and transparency and data sharing are all important steps to dealing with lack of publication (see Chap. 20).

In planning for a new trial, an investigator may want to obtain unpublished data from other investigators who have conducted trials in a similar population or tested the same intervention. Tables and figures in actual manuscripts seldom include everything that may be of interest. The situation is changing with online material available on journal websites. Many journals now publish full protocols, forms, manuals, and even raw data [21]. However, no uniform mechanism exists today for getting access to such study material from terminated trials. If information is available, it may not be in a reasonable and easily retrievable form. Substantial cooperation is usually required from the investigators originally involved in the data collection and analysis [22], and standards for data sharing and open access to trial data are evolving [23] (see Chap. 20).

The storage of biological material has raised new issues as it relates to genetic analyses. Biospecimens from well-characterized populations followed for long durations in clinical trials are in demand. These can be used to determine whether participant subgroups with a specific genotype are more likely to benefit or to experience serious adverse effects. The availability of these specimens for specific analysis depends on the wording of the informed consent (see Chap. 2). Patient privacy has to be considered, as always.

Storage of biomaterials may be costly. Freezers must be maintained, and a system for labeling and retrieving specimens or aliquots without damaging the remaining material must be implemented. Unlike with retrieval and distribution of data, many specimens may only be used once. Therefore, investigators need to develop a system for deciding when and how to use or distribute biospecimens. The cost and benefit, as well as the duration of storage must be considered. Central specimen repositories have been created to which investigators may be able to send their materials.

In summary, most trials collect an excess of study material, and it may not make sense to store everything. The investigator has to consider logistics, the length of the storage period, and cost. He also has to keep in mind that biological material, for example, deteriorates with time and laboratory methods change.