Empirical studies on the researcher degrees of freedom in biomedical data analysis

• Applicant: Professorin Dr. Anne-Laure Boulesteix
• Subject Area: Epidemiology and Medical Biometry/Statistics
• Term: since 2019
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 424874387

This project takes a metascientific perspective on statistical analyses performed as part of interventional or observational studies in health sciences. It is thus at the interface between meta-research, statistics and medicine. The four objectives of metascience are to (i) identify problems, (ii) investigate problems, (iii) develop solutions, and (iv) evaluate solutions. Our project focuses mainly on (ii) and (iii) and possibly includes components of (i) and (iv). The first project phase focused on the researcher degrees of freedom in statistical analyses. While some of the new subprojects (WP1.1 and 1.2) will extend the studies conducted in the first period towards providing solutions, further aspects of the researcher degrees of freedom will be explored as well, namely the degrees of freedom in relation to the design of biomedical studies (WP2.1 and 2.2) and the degrees of freedom in the interpretation of the results of given statistical analyses (WP3). The objective of WP1.1 is to develop a minP-type procedure for multiple testing adjustment in the context of the multiplicity of analysis strategies that does not rely on permutations. The objective of WP1.2 is to formalize, extend and empirically investigate the following informal rule we proposed previously: “Multiple testing should be adjusted for only in cases where authors put more weight on tests that yield significant results because of their significance when reporting, discussing, and interpreting their findings”. In WP2.1, we will empirically investigate and formalize the researcher degrees of freedom (uncertainties) that affect sample size calculations performed as part of the planning of medical studies, in order to provide tentative recommendations, while the aim of WP2.2 is to examine the feasibility of so-called adaptive designs beyond the context of clinical trials for addressing the uncertainty arising from the researcher degree of freedom. Finally, in WP3, we will consider the statistical analysis as fixed and investigate the degrees of freedom at work at the interpretation and visualization stage.

• DFG Programme: Research Grants