Sources of Bias in Randomised Controlled Trials

1. Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk www.rcts.org

2. Selection Bias - A reminder • Selection bias is one of the main threats to the internal validity of an experiment. • Selection bias occurs when participants are SELECTED for an intervention on the basis of a variable that is associated with outcome. • Randomisation or other similar methods abolishes selection bias.

3. After Randomisation • Once we have randomised participants we eliminate selection bias but the validity of the experiment can be threatened by other forms of bias, which we must guard against.

4. Forms of Bias • Subversion Bias • Technical Bias • Attrition Bias • Consent Bias • Ascertainment Bias • Dilution Bias • Recruitment Bias

5. Bias (cont) • Resentful demoralisation • Delay Bias • Chance Bias • Hawthorne effect • Analytical Bias.

6. Subversion Bias • Subversion Bias occurs when a researcher or clinician manipulates participant recruitment such that groups formed at baseline are NOT equivalent. • Anecdotal, or qualitative evidence (I.e gossip), suggest that this is a widespread phenomenon. • Statistically this has been demonstrated as having occurred widely.

7. Subversion - qualitative evidence • Schulz has described, anecdotally, a number of incidents of researchers subverting allocation by looking at sealed envelopes through x-ray lights. • Researchers have confessed to breaking open filing cabinets to obtain the randomisation code. Schulz JAMA 1995;274:1456.

8. Quantitative Evidence • Trials with adequate concealed allocation show different effect sizes, which would not happen if allocation wasn’t being subverted. • Trials using simple randomisation are too equivalent for it to have occurred by chance.

9. Poor concealment • Schulz et al. Examined 250 RCTs and classified them into having adequate concealment (where subversion was difficult), unclear, or inadequate where subversion was able to take place. • They found that badly concealed allocation led to increased effect sizes – showing CHEATING by researchers.

10. Comparison of concealment Schulz et al. JAMA 1995;273:408.

11. Case Study • Subversion is rarely reported for individual studies. • One study where it has been reported was for a large, multicentred surgical trial. • Participants were being randomised to 5+ centres using sealed envelopes.

12. Case-study (cont) • After several hundred participants had been allocated the study statistician noticed that there was an imbalance in age. • This age imbalance was occurring in 3 out of the 5 centres. • Independently 3 clinical researchers were subverting the allocation

13. Mean ages of groups

14. Example of Subversion

15. Concealment • Both the Schulz and Kjaergard considered sealed opaque envelopes to be ‘adequate’ measures of concealment. • Envelopes can be subverted by being opened in advance.

16. More Evidence • Hewitt and colleagues examined the association between p values and adequate concealment in 4 major medical journals. • Inadequate concealment largely used opaque envelopes. • The average p value for inadequately concealed trials was 0.022 compared with 0.052 for adequate trials (test for difference p = 0.045). Hewitt et al. BMJ;2005: March 10th.

17. More Examples • Berger has collected 30 case examples of potential subversion of the allocation process in clinical trials. • Because allocation subversion is scientific misconduct it is likely that there are many other, undetected, cases. Berger. Selection Bias and Covariate Imbalances in Randomized Clinical Trials 2005: Wiley, Chicester.

18. Recent Blocked Trial “This was a block randomised study (four patientsto each block) with separate randomisation at each of the threecentres. Blocks of four cards were produced, each containing twocards marked with "nurse" and two marked with "house officer."Each card was placed into an opaque envelope and the envelopesealed. The block was shuffled and, after shuffling, was placedin a box.” Kinley et al., BMJ 2002 325:1323.

19. What is wrong here? Kinley et al., BMJ 325:1323.

20. Problem? • If block randomisation of 4 were used then each centre should not be different by more than 2 patients in terms of group sizes. • Two centres had a numerical disparity of 11. Either blocks of 4 were not used or the sequence was not followed.

21. Restricted allocation and subversion • The drawback with any form of allocation restriction is that it allows some prediction. • Simple randomisation has no ‘memory’ of the previous allocation. In contrast, blocked allocation allows the probability of an allocation to be linked to the previous allocation. • Merely guessing that the next allocation will be the opposite of the previous one will result in a prediction more accurate than by chance. • This can, in theory, allow subversion.

22. Possible subversion • In a RCT of rehabilitation for the treatment of hip fracture gross baseline imbalances were detected favouring the control group. • Secure telephone allocation had been used. But blocked allocation, size 6, had been used. • Exploratory analysis of imbalances suggested partially successful prediction of block allocation. Turner J. 2002, Unpublished PhD Thesis, University of York.

23. Wither restricted allocation? • Simple randomisation followed by analysis of covariance (ANCOVA) is as efficient as restricted randomisation and ANCOVA for sample sizes > 50. • Restricted allocation increases risk of prediction and predictability. • For large trials simple allocation followed by ANCOVA reduces risk of prediction. Rosenberger WF, Lachin JM. Randomisation in clinical trials: Theory and practice. Wiley Interscience, 2002, John Wiley and Sons, New York.

24. Subversion - more evidence • In a survey of 25 researchers 4 admitted to keeping ‘a log’ of previous allocations to try and predict future allocations. Brown et al. Stats in Medicine, 2005,24:3715.

25. Testing for subversion • Comparison of baseline characteristics may help if subversion is suspected. Although this will only identify gross subversion. • If blocked allocation is used a statistical test – Bergner-Exner test, may help identify subversion.

26. Concealment: Recommendations • Allocation sequence must be independently generated and kept secret from the people who are enrolling participants. • A secure method of giving allocation to the recruiters must be developed, opaque envelopes are inadequate.

27. Subversion - summary • Appears to be widespread. • Secure allocation usually prevents this form of bias. • Need not be too expensive. • Essential to prevent cheating.

28. Secure allocation • Can be achieved using telephone allocation from a dedicated unit. • Can be achieved using independent person to undertake allocation.

29. Technical Bias • This occurs when the allocation system breaks down often due a computer fault. • A great example is the COMET I trial (COMET II was done because COMET 1 suffered bias).

30. COMET 1 • A trial of two types of epidural anaesthetics for women in labour. • The trial was using MIMINISATION via a computer programme. • The groups were minimised on age of mother and her ethnicity. • Programme had a fault. COMET Lancet 2001;358:19.

31. COMET 1 – Technical Bias

32. COMET II • This new study had to be undertaken and another 1000 women recruited and randomised. • LESSON – Always check the balance of your groups as you go along if computer allocation is being used.

33. Attrition Bias • Usually most trials lose participants after randomisation. This can cause bias, particularly if attrition differs between groups. • If a treatment has side-effects this may make drop outs higher among the less well participants, which can make a treatment appear to be effective when it is not.

34. Attrition Bias • We can avoid some of the problems with attrition bias by using Intention to Treat Analysis, where we keep as many of the patients in the study as possible even if they are no long ‘on treatment’.

35. Selection bias after randomisation • Selection bias is avoided if ALL participants who are randomised are completely followed up. • Often there is some attrition – after randomisation some refuse to continue to take part. • Or some may refuse the intervention but can still be tracked – IMPORTANT to distinguish between these.

36. What is wrong here?

37. Ascertainment Bias • This occurs when the person reporting the outcome can be biased. • A particular problem when outcomes are not ‘objective’ and there is uncertainty as to whether an event has occurred. • Example, of homeopathy study of histamine, showed an effect when researchers were not blind to the allocation but no effect when they were. • Multiple sclerosis treatment appeared to be effective when clinicians unblinded but ineffective when blinded.

38. Resentful Demoralisation • This can occur when participants are randomised to treatment they do not want. • This may lead to them reporting outcomes badly in ‘revenge’. • This can lead to bias.

39. Resentful Demoralisation • One solution is to use a patient preference design where only participants who are ‘indifferent’ to the treatment they receive are allocated. • This should remove its effects.

40. Hawthorne Effect • This is an effect that occurs by being part of the study rather than the treatment. Interventions that require more TLC than controls could show an effect due to the TLC than the drug or surgical procedure. • Placebos largely eliminate this or TLC should be given to controls as well.

41. Analytical Bias • Once a trial has been completed and data gathered in it is still possible to arrive at the wrong conclusions by analysing the data incorrectly. • Most IMPORTANT is ITT. • Also inappropriate sub-group analyses is a common practice.

42. Intention To Treat • Main analysis of data must be by groups as randomised. Per protocol or active treatment analysis can lead to a biased result. • Those patients not taking the full treatment are usually quite different to those that are and restricting the analysis can lead to bias.

43. Sub-Group Analyses • Once the main analysis has been completed it is tempting to look to see if the effect differs by group. • Is treatment more or less effective in women? • Is it better or worse among older people? • Is treatment better among people at greater risk?

44. Sub-Groups • All of these are legitimate questions. The problem is the more subgroups one looks at the greater is the chance of finding a spurious effect. • Sample size estimations and statistical tests are based on 1 comparison only.

45. Sub-Group and example. • In a large RCT of asprin for myocardial infarction a sub-group analysis showed that people with the star signs Gemini and Libra aspirin was INEFFECTIVE. • This is complete NONSENSE! • This shows dangers of subgroup analyses. Lancet 1988;ii:349-60.

46. Sub groups • To avoid spurious findings these should be pre-specified and based on a reasonable hypothesis. • Pre-specification is important avoid data dredging as if you torture the data enough it will confess.

47. Summary • Despite the RCT being the BEST research method unless expertly used it can lead to biased results. • Care must be taken to avoid as many biases as possible.