12. Assessment and Reporting of Harm

There are four distinct advantages to assessment of harm in clinical trials, as opposed to other kinds of clinical research. First, adverse events can be defined prospectively, which allows proper hypothesis testing and adds substantial credibility. Post hoc observations, common in the area of harm, are often difficult to interpret in terms of causation and therefore often lead to controversy.

Second, randomized clinical trials by definition have a proper and balanced control group which allows for comparisons between the study groups. Randomization assures that intervention and control groups have similar characteristics—even those unknown to science at the time the trial was conceived. Other study designs have a dilemma when comparing users of a particular intervention to non-users. In observational studies, there is no guarantee that the user and non-user groups are comparable. There are clinical reasons why some people are prescribed a particular intervention while others are not. Observed group differences can be intervention-induced, due to differences in the composition and characteristics of the groups, or a combination thereof. Statistical adjustments can help but will never be able to control fully for unmeasured differences between users and non-users.

Third, clinical trials with a blinded design reduce potential biases in the collection, assessment and reporting of data on harm (Chap. 7).

Fourth, participants in clinical trials are closely and systematically assessed, including physical examinations, regular blood work, weekly or monthly clinic visits, vital signs, clinical events, and detailed assessment of concomitant medications.

There are also four potential limitations in relying on clinical trials for evaluation of harm. First, the trial participants are a selected non-random sample of people with a given condition who volunteered for the trial. The selectivity is defined by the scope of the trial inclusion and exclusion criteria and the effects of enrolling only volunteers. In general, trial participants are healthier than non-participants with the same disease. In addition, certain population groups may be excluded, for example, women who are pregnant or breastfeeding. Trials conducted prior to regulatory agency approval of a product are typically designed to document clear findings of benefit and, therefore, often exclude from participation those who are old, have complicating medical conditions and/or are taking other medications which may affect the outcome. Trial sponsors also exclude participants at higher risk of suffering an adverse event. This reduces the incidence of such events and contributes to the likelihood of not documenting harm. The absence of serious adverse effects observed in low-risk participants in pre-approval trials is no assurance that a drug lacks harmful effects when it reaches the marketplace. Another limitation is that the ascertainment of adverse events often relies on volunteered information by the participant rather than specific, solicited information (see below). An early survey showed that most FDA-approved drugs have one serious adverse effect detected after approval when there is more exposure to higher-risk patients and longer treatment duration [3]. More recent high-profile cases of serious adverse effects not detected pre-approval are the treatments of osteoarthritis with COX-2 inhibitors [4–7], of type II diabetes with rosiglitazone [8–10], and prevention of thromboembolic events with oral anticoagulants [11–13]. The reported high rates of new Boxed Warnings and drug withdrawals over the past two decades illustrate a limitation of FDA’s current process for documenting real and potential harm pre-approval [14].

A second limitation relates to the statistical power of finding a harm, if it exists. Small sample sizes and short trial durations, as well as the focus on low-risk populations, reduce the likelihood of detecting serious adverse effects. Drug manufacturers often conduct a large number of small, short-term trials, and their large trials are often not of long duration. Due to limited statistical power, clinical trials are often unreliable for attributing causality to rare serious adverse events. Approximately 3,000 participants are required to detect a single case with 95% probability, if the true incidence is one in 1,000; a total of 6,500 participants are needed to detect three cases [15]. When a new drug is approved for marketing, approximately 500–2,000 participants have typically been exposed to it in both controlled and uncontrolled settings. More commonly, rare serious adverse effects are initially discovered through case reports, other observational studies or reports of adverse events filed with regulatory agencies after approval [16, 17]. However, clinical trials can detect precursors of serious adverse effects through measurements such as elevated ALT levels (acute liver failure) or prolonged QT interval on the electrocardiogram (sudden cardiac death). Vandenbroucke and Psaty [18] properly concluded that “the benefit side [of drugs] rests on data from randomized trials and the harms side on a mixture of randomized trials and observational evidence, often mainly the latter.”

Third, inability to detect late serious adverse effects is another potential limitation of clinical trials. When a new compound is introduced for long-term treatment of a non-life threatening disease, the minimum regulatory standard is only several hundred participants exposed for 1 year or longer [19]. This is obviously inadequate for evaluation of drugs intended for chronic or long-term use. Moreover, a long lag time to harm must be considered for drugs that may be carcinogenic or have adverse metabolic effects. For example, the lag time for carcinogens to cause cancer may often be longer than most long-term trials. We support the view that evaluation of harm should continue the entire time a drug intended for chronic use is on the market [20].

Fourth, the investigators or sponsors may be unaware of some adverse effects because they are unexpected, or, in the case of known adverse effects, not ascertained. Potentially lethal cardiac rhythm disturbances may not be identified because electrocardiographic studies are not performed. Diabetes risk may be overlooked because laboratory testing does not include periodic assessment of Hb_A1c. Adverse effects related to sexual function or suicidal ideation may be underestimated because participants rarely volunteer information about sexual problems or suicidal ideation in response to general questions about changes in their health status. Ascertaining withdrawal and rebound effects require a special protocol to monitor discontinuation symptoms. Drug interactions may be overlooked because of rigid exclusion criteria in the protocol and failure to analyze concomitant medication data in relation to adverse events. Additionally, it is very challenging to be rigorous in these analyses.

The methods for collecting information on harm should take advantage of the strengths of clinical trials and to supplement them with properly designed and conducted observational studies post-trial, especially if issues or signals of harm emerge. Establishment of such long-term safety registries as one tool for post-marketing surveillance is becoming more common [21].

As pointed out earlier in this chapter, randomized clinical trials are not optimal for the detection of rare, late and unexpected serious adverse events. Experience has shown that critical information on serious reactions comes from multiple sources.

The role of clinical trials in identifying serious adverse reactions was investigated in an early study by Venning [16], who reviewed the identification and report of 18 adverse reactions in a variety of drugs. Clinical trials played a key role in identifying only three of the 18 adverse effects discussed. Another comparison of evidence of harm of various interventions in 15 large randomized and observational studies showed that the non-randomized studies often were more likely to find adverse effects [22].

A clinical trial may, however, suggest that further research on adverse reactions would be worthwhile. As a result of implications from the Multiple Risk Factor Intervention Trial [23] that high doses of thiazide diuretics might increase the incidence of sudden cardiac death, Siscovick and colleagues conducted a population-based case-control study [24]. This study confirmed that high doses of thiazide diuretics, as opposed to low doses, were associated with a higher rate of cardiac arrest.

Drugs of the same class generally are expected to have a similar effect on the primary clinical outcome of interest. However, they may differ in degree if not in kind of adverse effects. One illustration is cerivastatin which was much more likely to cause rhabdomyolysis than the other marketed statins [25]. Longer acting preparations, or preparations that are absorbed or metabolized differently, may be administered in different doses and have greater or lesser adverse effects. It cannot be assumed in the absence of appropriate comparisons that the adverse effects from similar drugs are or are not alike. As noted, however, a clinical trial may not be the best vehicle for detecting these differences, unless it is sufficiently large and of long duration.

Genomic biomarkers have assumed an increasing and important role in identifying people at an increased risk of adverse effects from medications. A large number of FDA approved drugs have pharmacogenomics information in different sections of the labeling [26]. Thus, adverse drugs effects observed in genetically defined subgroups of people are reflected in label additions of Boxed Warnings, Contraindications, Warnings, Precautions and Drug Interactions.

The rationale for defining adverse events in the protocol is similar to that for defining any important benefit variable; it enables investigators to record something in a consistent manner. Further, it allows someone reviewing a trial to assess it more accurately, and possibly to compare the results with those of other trials of similar interventions.

Because adverse events are typically viewed as secondary or tertiary response variables, they are not often systematically and prospectively evaluated and given the same degree of attention as the primary and secondary benefit endpoints. They usually are not defined, except by the way investigators apply them in their daily practice. A useful source is the Investigator’s Brochure for the study drug. The diagnosis of acute myocardial infarction may be based on non-standardized hospital records. Depression may rely on a patient-reported symptom of non-specified severity and duration rather than a careful evaluation by a psychiatrist or responses to a standardized depression questionnaire. Thus, study protocols seldom contain written definitions of adverse events, except for those that are recognized clinical conditions. Multicenter trials open the door to even greater levels of variability in event definitions. In those cases, an adverse event may be simply what each investigator declares it to be. Thus, intrastudy consistency may be as poor as interstudy consistency.

However, given the large number of possible adverse events, it is not feasible to define all of them in advance and, in addition, many do not lend themselves to satisfactory definition. Some adverse events cannot be defined because they are not listed in advance, but are spontaneously mentioned by the participants. Though it is not always easy, important adverse events which are associated with individual signs or laboratory findings, or a constellation of related signs, symptoms, and laboratory results can and should be well-defined. These include the events known to be associated with the intervention and which are clinically important, i.e. adverse events of special interest. Other adverse events that are purely based on a participant’s report of symptoms may be important, but are more difficult to define. These may include nausea, fatigue, or headache. Changes in the degree of severity of any symptom should be part of the definition of an adverse event. The methods by which adverse events were ascertained should be stated in any trial publication.

The simplest way of recording presence of an adverse event is with a yes/no answer. This information is likely to be adequate if the adverse event is a serious clinical event such as a stroke, a hospitalization or a significant laboratory abnormality. However, symptoms have other important dimensions such as severity, duration and frequency of recurrence.

The severity of subjective symptoms is typically rated as mild, moderate or severe. However, the clinical relevance of this rating is unclear. Participants have different thresholds for perceiving and reporting their symptoms. In addition, staff’s recorded rating of the reported symptom may also vary. One way of dealing with this dilemma is to consider the number of participants who were taken off the study medication due to an adverse event, the number who had their dose of the study medication reduced and those who continued treatment according to protocol in spite of a reported adverse event. This classification of severity makes clinical sense and is generally accepted. A challenge may be to decide how to classify participants who temporarily are withdrawn from study medication or have their doses temporarily reduced.

The duration or frequency with which a particular adverse event occurs in a participant can be viewed as another measure of severity. For example, episodes of nausea sustained for weeks rather than occasionally is a greater safety concern. Investigators should plan in advance how to assess and present all severity results.

The duration of a trial has a substantial impact on adverse event assessment. The longer the trial, the more opportunity one has to discover adverse events, especially those with low frequency. Also, the cumulative number of participants in the intervention group complaining will increase, giving a better estimate of the incidence of the adverse event. Of course, eventually, most participants will report some general complaint, such as headache or fatigue. However, this will occur in the control group as well. Therefore, if a trial lasts for several years, and an adverse event is analyzed simply on the basis of cumulative number of participants suffering from it, the results may not be very informative, unless controlled for severity and recurrences. For example, the incidence could be annualized in long-term trials.

Duration of follow-up is also important in that exposure time may be critical. Some drugs may not cause certain adverse effects until a person has been taking them for a minimum period. An example is the lupus syndrome with procainamide [32]. Given enough time, a large proportion of participants will develop this syndrome, but very few will do so if treated for only several weeks. Other sorts of time patterns may be important as well. Many adverse effects even occur soon after initiation of treatment. In such circumstance, it is useful, and indeed prudent, to monitor carefully participants for the first few hours or days. If no effects occur, the participant may be presumed to be at a low risk of developing these effects subsequently.

In the Diabetes Control and Complications Trial (DCCT) [33], cotton exudates were noted in the eyes early after onset of the intervention of the participants receiving tight control of the glucose level. Subsequently, the progression of retinopathy in the regular control group surpassed that in the tight control group, and tight control was shown to reduce this retinal complication in insulin-dependent diabetes. Focus on only this short-term adverse effect might have led to early trial termination. Fortunately, DCCT continued and reported a favorable long-term benefit-harm balance.

Figure 12.1 illustrates the first occurrence of ulcer symptoms and complaints of stomach pain, over time, in the Aspirin Myocardial Infarction Study [30]. Ulcer symptoms rose fairly steadily in both the aspirin and placebo groups, peaking at 36 months. In contrast, complaints of stomach pain were maximal early in the aspirin group, then decreased. Participants on placebo had a constant, low level of stomach pain complaints. If a researcher tried to compare adverse effects in two studies of aspirin, one lasting weeks and the other several months, the findings would be different. To add to the complexity, the aspirin data in a study of longer duration may be confounded by changes in aspirin dosage and concomitant therapy.

An intervention may cause continued discomfort throughout a trial, and its persistence may be an important feature. Yet, unless the discomfort is considerable, such that the intervention is stopped, the participant may eventually stop complaining about it. Unless the investigator is alert to this possibility, the proportion of participants with symptoms at the final assessment in a long-term trial may be misleadingly low.

The most basic form of assessment of harm is a complete accounting for all participants including those who did not complete the trial. Overall dropout rates are a useful measure of the tolerability of the drug or other interventions, and can be compared across many interventions. Dropout reporting is typically divided into at least three subcategories: dropout due to adverse events, dropouts for lack of efficacy and dropouts for administrative reasons. Assignment of a case to these subcategories may be more subjective than it appears. Lack of efficacy dropouts may rise because symptomatic adverse events might persuade some participants that they are not getting enough benefit to continue. Withdrawals of consent or other administrative departures may conceal problems with the drug, or the conduct of the trial. The overall dropout rate across all categories should be presented. If the dropouts have characteristics over time (such as dropouts related to short-term, early onset adverse events), some form of survival analysis of dropout rate over time may provide useful insights for managing treatment or suggest a need for dose titration.

Another standard analysis consists of a table of reported adverse events at the MedDRA level of Preferred Terms, with control and each intervention arm forming a column for easy comparison across groups. To make the list manageable in length, investigators typically set a threshold value for a subset of adverse events that total more than 1%, 5%, or 10% of patients. This has the major drawback of excluding less common adverse events which may be the more serious ones. Tests of statistical significance may be presented, but must be interpreted cautiously. Longer tables are usually organized by body system using the MedDRA System Organ Class. These standard event tables do not distinguish the severity and frequency of adverse events and are typically dominated by frequently occurring symptoms such as headache, nausea or dizziness.

Standard safety analysis may also include a listing of deaths, serious adverse events, clinically significant laboratory abnormalities, and changes in vital signs.

Possible adverse effects that could be reasonably expected from the known mechanism of action of the evaluated intervention, prior studies, or underlying participant conditions could be defined and analyzed from the perspectives of ascertainment, classification, and in particular statistical power, but these are rarely done. An investigator needs to consider prospectively and in the analysis the possibility of Type I or Type II error in the context of all three.

Adjudication is a tool frequently used when adverse events are of particular importance or are difficult to define. Adjudicated events are typically assessed by expert panels blinded to study group and following written protocols. While adjudicated results are typically seen as increasing the credibility and objectivity of the findings they may also reduce already limited statistical power by discarding cases with incomplete information. Adjudication can also be abused to suppress adverse event counts though unreasonably specific and restrictive case definitions. In addition, bias may be introduced if the adjudicators are not fully blinded. In the Randomized Evaluation of Long-Term Anticoagulation Therapy (RE-LY) trial, a team of adjudicators reviewed the outcome documents after reported to have been blinded [11]. Subsequently, the FDA took a closer look at the documents and concluded that information on intervention group assignment was available in 17% of the cases [34]. The credibility of adjudication results can be enhanced by accounting for possible but excluded cases.

All post hoc analyses of adverse events may be subject to the criticism that it introduces bias because the analyses were not prospectively defined. Bias may also be introduced by problems of ascertainment and classification. These concerns are valid, but must be considered in light of two factors. First, analyses of prespecified events may themselves have biases and additional, even post hoc, analyses may provide further insight. Second, good clinical research is expensive, difficult to conduct, and seldom repeated without addressing new scientific issues. Therefore, post hoc analysis may yield important information and clues not otherwise obtainable.

One straightforward post hoc analysis addresses limitations of adverse event classification that occur due to the underlying MedDRA terminology. With approximately 20,000 Preferred Terms to describe an adverse event, this terminology permits substantial precision at the cost of disaggregating adverse events and is raising issues about accuracy. For example, at the Preferred Term level, a case of depression could be coded into any of 22 different terms (Table 12.2). Problems of gastrointestinal tolerability might be divided into nausea, vomiting, dyspepsia, and various forms of abdominal pains. Adverse event tables can be examined at all three key levels of the MedDRA hierarchy (Preferred, High Level and High Level Group Terms) as well through other categories created or Standardized MedDRA Queries. Additional understanding of adverse events could be expanded through examining time to reaction, effect duration, or severity. While these post hoc analyses may provide valuable insights into the harm of drugs and medical interventions, they should be specifically identified as separate from prospectively defined analyses.

Statistical techniques for data mining may provide additional opportunities to detect new signals of harm overlooked by clinical investigators in blinded trials. These techniques can be applied initially to the analysis of spontaneous adverse event reports but can be used both for signal detection in individual clinical trials and pooled data sets. With large populations, repeated visits, multiple outcome measures, many concomitant medications, and measures of underlying disease severity, the accumulated data are often too massive to exploit effectively with a prospective data analysis plan. However, the results of data mining analysis should be regarded as hypothesis generating that, after evaluation, would require additional investigation. Such signals may provide a useful basis for additional post hoc studies of existing data or enable prespecified analysis in future clinical trials. Data mining results may also provide context and focus to interpret particular results that were prespecified. Statistical tools such as the false discovery rate estimation [35] can help identify reliable associations in larger spontaneous reporting databases; other analysis might point to the need to explore associations that appeared borderline initially.

When individual trials are inconclusive, one approach is the combination of data on harm from multiple trials in a meta-analysis or systematic review (see Chap. 18).

Meta-analyses or pooled analyses conducted by manufacturers are commonly included in New Drug Applications submitted to regulatory agencies. Meta-analyses of treatment harm are now being published in leading medical journals. Singh and colleagues published three meta-analyses showing that rosiglitazone and pioglitazone double the risk of heart failure and fractures (in women) in type 2 diabetes [36, 37] and that rosiglitazone, in contrast to pioglitazone, also increases the risk of heart attacks [38]. None of these adverse effects was recognized at the time of regulatory approval of these drugs. Singh and colleagues concluded that cumulative clinical trial data revealed increased cardiovascular harm associated with rofecoxib a couple of years before the drug was withdrawn from the U.S. market. It has been recommended that cumulative meta-analysis be conducted to explore whether and when pooled adverse effect data reveal increased harm [39].

It is important to keep in mind that meta-analyses of harm have many limitations. Adverse event data in published studies are usually limited and event ascertainment seldom disclosed. Individual trials revealing unfavorable results may never be reported or published leading to publication bias and underestimation of the true rate of adverse effects. Experience has shown that conclusions from meta-analyses of a large number of small trials are not always confirmed in subsequent large trials.

Even though the clinical trials were prospectively designed, meta-analysis for harm is vulnerable to all the biases of a post hoc study design about a controversial safety issue when both the relevant trials and the number of events in each trial are already known by the study investigators. Small differences in inclusion or exclusion criteria can have large effects on the relative risk calculation, but are not evident in published results.

A substantial problem arises when investigators report that a meta-analysis of numerous trials detected no evidence of an adverse drug event reported using other methods. The failure to disprove the null hypothesis (no difference observed) is then claimed to be an assurance of safety. In this setting, additional evidence is required to rule out a simple Type II statistical error—that a difference existed but could not be detected in this study. In comparative clinical trials with an active drug control this problem is managed with relatively rigorous statistical standards for demonstrating non-inferiority. No such standards exist for meta-analysis of drug adverse events. Finally, when the magnitude of reported harm is small (for example a relative risk <2) all these imperfections in this technique mandate caution in interpreting the results.