Does the choice of estimator change the pooled estimate?

Slightly, through the weights, but the larger effect is on the confidence interval. A meta-analysis whose conclusion flips between estimators was never robust, and that instability is itself worth reporting rather than resolving by picking one.

Should I report tau² or tau?

Report tau² for completeness and tau alongside it, since tau is on the scale of the effect measure and therefore interpretable. For a standardised mean difference, a tau of 0.30 tells a reader that true effects vary by roughly a third of a standard deviation between studies, which is more informative than tau² = 0.09.

With only three studies, is a random-effects model appropriate?

Tau² cannot be estimated reliably from three studies by any method. Options are to present a fixed-effect analysis with an explicit statement of the assumption, to use a random-effects model with an informative prior in a Bayesian framework, or to describe the studies narratively. Whichever you choose, do not present a three-study random-effects interval as though tau² were well estimated.

Tau² Estimators and Why the Default Is Not Always Right

DerSimonian-Laird is the default in most software and performs poorly with few studies. What the alternatives do, and why the Hartung-Knapp adjustment matters.

20 September 20264 min

Every random-effects meta-analysis estimates tau², the variance of true effects between studies. Tau² determines the study weights, the width of the confidence interval, and the prediction interval.

Most software estimates it by the DerSimonian-Laird method, because that is what the default has been since 1986. It is not the best available option, and with few studies it is not a good one.

What goes wrong with the default

DerSimonian-Laird is a method-of-moments estimator. It is fast, it has a closed form, and it tends to underestimate tau² when the number of studies is small or heterogeneity is substantial.

Underestimating tau² makes confidence intervals too narrow, which means false positive rates above the nominal 5%. Meta-analyses of five to ten studies, which is most of them, are exactly where the problem bites.

Veroniki and colleagues (2016) reviewed the available estimators and their uncertainty, and Langan and colleagues (2019) compared their performance in simulation. The broad conclusion is that restricted maximum likelihood performs well across most realistic scenarios and is a sensible default for continuous outcomes, with the Paule-Mandel estimator performing well for dichotomous outcomes.

The adjustment that matters more

Changing the estimator helps. Changing how the confidence interval around the pooled effect is constructed helps more.

The Hartung-Knapp-Sidik-Jonkman adjustment uses a t-distribution rather than a normal distribution and accounts for the uncertainty in the estimate of tau². IntHout, Ioannidis and Borm (2014) found it substantially outperforms the standard approach, particularly with few studies, producing intervals with coverage closer to nominal.

It usually widens the interval. That is the point: the standard method is overconfident, and a wider interval that is correctly calibrated is more useful than a narrow one that is not.

One caveat. With very few studies and little heterogeneity, the adjustment can occasionally produce an interval narrower than the standard one, which some methodologists regard as anomalous. Sensitivity analysis using both approaches settles it.

What to prespecify

A protocol sentence covering both choices:

"Random-effects meta-analysis will use the restricted maximum likelihood estimator for tau², with confidence intervals around the pooled effect calculated using the Hartung-Knapp-Sidik-Jonkman adjustment. Results using the DerSimonian-Laird estimator will be reported as a sensitivity analysis."

Reporting both is worth the extra column. Where they agree, the conclusion is robust to a choice most readers will not have considered. Where they disagree, that is a finding about how fragile the evidence is.

Implementation

In R, metafor's rma() takes method = "REML" as its default and test = "knha" for the adjustment. The meta package uses method.tau and hakn = TRUE. Stata's meta suite supports both. RevMan implements DerSimonian-Laird for random-effects analyses, which is worth knowing if your review is Cochrane-registered and a reviewer asks for REML.

Whatever you use, name it in the methods. "A random-effects model was used" is not a specification, and a statistical reviewer will ask which estimator and which interval method.

References

DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7(3), 177–188. https://doi.org/10.1016/0197-2456(86)90046-290046-2)

IntHout, J., Ioannidis, J. P. A., & Borm, G. F. (2014). The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Medical Research Methodology, 14, 25. https://doi.org/10.1186/1471-2288-14-25

Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M. (2019). A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. Research Synthesis Methods, 10(1), 83–98. https://doi.org/10.1002/jrsm.1316

Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P. T., Langan, D., & Salanti, G. (2016). Methods to estimate the between-study variance and its uncertainty in meta-analysis. Research Synthesis Methods, 7(1), 55–79. https://doi.org/10.1002/jrsm.1164

Common questions

Does the choice of estimator change the pooled estimate?: Slightly, through the weights, but the larger effect is on the confidence interval. A meta-analysis whose conclusion flips between estimators was never robust, and that instability is itself worth reporting rather than resolving by picking one.
Should I report tau² or tau?: Report tau² for completeness and tau alongside it, since tau is on the scale of the effect measure and therefore interpretable. For a standardised mean difference, a tau of 0.30 tells a reader that true effects vary by roughly a third of a standard deviation between studies, which is more informative than tau² = 0.09.
With only three studies, is a random-effects model appropriate?: Tau² cannot be estimated reliably from three studies by any method. Options are to present a fixed-effect analysis with an explicit statement of the assumption, to use a random-effects model with an informative prior in a Bayesian framework, or to describe the studies narratively. Whichever you choose, do not present a three-study random-effects interval as though tau² were well estimated.