When Meta-Regression Needs More Studies Than You Have

Meta-regression relates study-level characteristics to effect size. It is the right tool for examining whether a continuous modifier, such as mean age, intervention duration, or baseline severity, explains heterogeneity, and it is better than dichotomising that variable into subgroups at an arbitrary cut point.

It also fails quietly when there are too few studies, and the failure looks like a result.

The ten-per-covariate rule

The working guidance, from Cochrane and from Thompson and Higgins (2002), is a minimum of around ten studies per covariate examined.

A meta-analysis of twelve studies supports one covariate. Not three. Fitting three covariates to twelve studies produces coefficients with intervals so wide that any conclusion drawn is an artefact of noise, and the model will nonetheless return numbers that look like findings.

The rule is a floor, not a target. Ten studies per covariate is the point at which the analysis becomes interpretable at all, not the point at which it becomes reliable.

Aggregation bias

Meta-regression uses study-level averages. A relationship between mean age and effect size across studies does not establish that older individuals respond differently, because the study-level relationship can differ in magnitude and even in direction from the individual-level one.

This is ecological fallacy applied to synthesis, and it is the limitation to state explicitly whenever a meta-regression finding is discussed. The honest phrasing is that studies with older participants on average reported larger effects, not that older participants benefit more.

The way around it is individual participant data, which is a different and much larger project.

Multiplicity, again

Each covariate tested is a hypothesis test. Testing six covariates across fifteen studies is a fishing expedition with a respectable name.

Prespecify the covariates in the protocol, keep the number small, and report every one tested including those that found nothing. A meta-regression reported only when significant is selective reporting inside your own analysis.

Where several covariates are examined, consider a permutation test, which adjusts for multiplicity and is implemented in metafor's permutest().

Reporting a model properly

Four elements, and reviews routinely omit two of them.

The coefficient with its confidence interval, on the scale of the effect measure, with a statement of what a one-unit change in the covariate corresponds to.

The proportion of between-study variance explained, R² analogue, and the residual tau². A model explaining a small fraction of substantial heterogeneity has not accounted for the variation, whatever the p-value on the coefficient says.

The number of studies contributing, which is often fewer than the total because covariates are frequently unreported in some trials.

A bubble plot. It shows the relationship, the precision of each study, and whether one influential outlier is driving the slope, which a table cannot.

A worked example

Eighteen trials of a supervised exercise programme, meta-regression of effect size on programme duration in weeks.

Coefficient −0.014 per week, 95% CI −0.031 to 0.003, R² analogue 11%, residual tau² 0.08 against 0.09 unadjusted, 15 studies reporting duration.

The reportable conclusion: longer programmes were associated with slightly larger effects, the association was not statistically significant, the model explained about a tenth of the between-study variance, and substantial heterogeneity remained unexplained. The finding is consistent with a dose-response relationship but does not establish one, and it operates at the study level rather than the participant level.

That is what a null meta-regression looks like written honestly, and it belongs in the paper.

References

Thompson, S. G., & Higgins, J. P. T. (2002). How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine, 21(11), 1559–1573. https://doi.org/10.1002/sim.1187