The randomised controlled trial has become an absolute requirement for demonstrating efficacy of medicines. That is quite right, as we have learned to our cost that studies without a randomised design often, or usually, mislead.
The trouble is that RCTs can also mislead, especially in the area of pain. And, indeed, there is quite a long list of things that can affect the estimation of efficacy. Four details can be fatal, in the sense of examples demonstrating that these characteristics have the ability to completely change results from positive to negative:
1. Trials that are NOT randomised.
2. Trials that are NOT blind.
3. Trials using last observation carried forward imputation when there is a high rate of adverse event withdrawals.
4. Trials that are small, where random effects and bias are real problems.
Of course, there are all sorts of other problems. Publication bias, or potential for publication bias, can be a problem when effect sizes are small, or the total amount of data are small, and especially a combination of these. But perhaps the largest issue is one of outcomes. In pain we too often in the past report averages – particularly average change in pain. The implicit assumption is that the average is the experience of most. Usually it is the experience of the least number, as pain responses tend to be all or nothing. And then there is the problem of patient selection.