Testing for blinding in sham-controlled studies for procedural interventions: the third-party video method

• For practitioners investigating procedural or surgical interventions performed to improve subjective outcomes, sham controls are vital.

• Video surveillance to assess blinding in sham-controlled randomized trials of procedural interventions is feasible and has potential to strengthen the validity of results and better differentiate a placebo effect of an intervention from a true effect.

• By video-recording procedures (without a direct view of the procedural site or critical step) and asking a professional familiar with the procedure to guess the treatment (experimental v. sham), investigators can assess whether the sham control was a convincing imitation of the active treatment.

• Assessing blinding through video surveillance would not only strengthen validity of the measured effects of procedure, but also offer a tool for interpreting results, asking investigators to reflect further on their methods and explain why observers were aware of intervention allocation.

When it comes to procedural or surgical interventions performed to improve subjective outcomes (e.g., joint pain, dyspnea, angina pectoris), sham controls are vital. Their use has clarified that perceived treatment effects from internal mammary ligation,1 partial meniscectomy,2 vertebroplasty3 and knee joint débridement4 are largely placebo effects. Recently, the Objective Randomised Blinded Investigation with Optimal Medical Therapy of Angioplasty in Stable Angina (ORBITA) study reached the surprising conclusion that exercise time was not significantly improved when patients received a stent for single-vessel angina, compared with a sham intervention.5 The trial found a nonsignificant 16-second difference in outcome favouring the stenting group, which was considered to be not clinically meaningful. Yet even this nonsignificant difference may be an artifact of a small, residual difference in blinding. When procedural sham trials find differences, how can we separate true treatment effects from inadequate or incomplete shams? Here, we suggest a method that may strengthen methodology and help to assess blinding: video surveillance during both active and sham procedures.

Practitioners attempt sham controls when they concede that the role of perception may be prominent in the active intervention in a randomized study — i.e., the possibility of placebo effect forms the justification for the sham — and a sham procedure is deemed ethical. In sham-controlled studies, both intervention and control groups are led to believe that they undergo the intervention, although the critical intervention step is performed only in the intervention group. Sham procedures must emulate the active treatment in characteristics such as length of time, preparation and practitioner demeanour. However, there is potential for unblinding during sham procedures that may not be readily apparent, despite a detailed description of design and methods. For instance, patients in the intervention arm may spend more time on the procedure table than those in the control arm. If conscious sedation is used — which is common in sham-controlled trials — participants’ awareness of a short procedure time may lead them to guess that they are in the control group. Moreover, merely asking participants about whether they underwent the procedure is an insufficient strategy to assess blinding. An intervention such as stenting may have an immediate salutary effect — thus, patients may know that they received the procedure, not because the blinding was ineffective, but rather as a result of an effective intervention.

Video surveillance during active and sham procedures is a way to further validate the effectiveness of blinding. By video-recording procedures (without a direct view of the procedural site or critical step) and asking a professional familiar with the procedure to guess the treatment (experimental v. sham), investigators can assess whether the sham control was a convincing imitation of the active treatment. Professionals blinded to treatment allocation may be able to judge the quality of a sham procedure in a way that unblinded clinicians performing the procedures may not. Evaluating the guesses of the third-party, blinded professionals alongside guesses of participants can determine the correct interpretation of the study, as shown in Figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1:

Four categories of blinding efficacy in sham trials assessed using video and questionnaire tools, based on the guesses regarding treatment type from the patients and third-party professional reviewers.

If patients are able to guess their treatment accurately but blinded professionals cannot, this signifies that blinding was effective and the reason for the participants’ correct guesses was likely a result of effective treatment. If both patients and blinded professionals correctly guess treatment type, it is likely that the sham procedure did not mimic the treatment group’s procedure successfully, and blinding was compromised. If both groups guess poorly, blinding is shown to be effective, and the intervention ineffective. Lastly, if the patients are inaccurate in their guesses but blinded professionals guess correctly, it is likely that the intervention does not work, despite an imperfect sham trial.

Assessing blinding through video surveillance would not only strengthen validity of the measured effects of procedure, but would also offer a tool for interpreting results, asking investigators to reflect further on their methods and explain why observers were aware of intervention allocation. For instance, video-recording the procedures used in the ORBITA trial would have allowed investigators to consider whether the observed, nonsignificant difference was a real effect, or merely an effect of subtle cues given to some participants, which helped them guess their group allocation.

The Vertebroplasty Versus Sham Procedure for Painful Acute Osteoporotic Vertebral Compression Fractures (VERTOS IV) study compared percutaneous vertebroplasty to a sham procedure in patients with acute osteoporotic vertebral compression fractures and found that the procedure had no greater effect on reducing pain at 12-month follow-up.3 Participants, internists and outcome assessors were blinded. After surgery, 82% of participants in the vertebroplasty group and 81% in the sham group believed they had received cement. In their discussion of the findings, the study’s authors compared their blinding success with 2 similar trials that also assessed blinding. The Investigational Vertebroplasty Safety and Efficacy Trial (INVEST)6 and Vertebroplasty for Acute Painful Osteoporotic Fractures (VAPOUR)7 trials both used sham procedures to determine vertebroplasty efficacy and found that 37% and 46% of the participants in the sham procedure group believed they had undergone cementation or were unsure. Assessing blinding of previous trials and fully reporting their blinding strategy allowed the VERTOS IV study team to strengthen the validity of their results. However, this method does not separate the possibility of a real difference in treatment effect from the possibility of more effective sham techniques. Adding video surveillance to the assessment would offer an additional tool to determine the discrepancies between sham procedures in each study and permit authors to suggest improvements in sham methods for future study investigators.

Video surveillance to assess blinding in sham-controlled randomized trials of procedural interventions is feasible and has potential to strengthen the validity of results and better differentiate a placebo effect of an intervention from a true effect. It should be noted that patients must consent to video surveillance and be informed as to why this approach can improve such research. If carefully done, blinding assessment through third-party video review can only strengthen such trials.