Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Elliot Gould; Hannah S Fraser; Timothy H Parker; Shinichi Nakagawa; Simon C Griffith; Peter A Vesk; Fiona Fidler; Daniel G Hamilton; Robin N Abbey-Lee; Jessica K Abbott; Luis A Aguirre; Carles Alcaraz; Irith Aloni; Drew Altschul; Kunal Arekar; Jeff W Atkins; Joe Atkinson; Christopher M Baker; Meghan Barrett; Kristian Bell; Suleiman Kehinde Bello; Iván Beltrán; Bernd J Berauer; Michael Grant Bertram; Peter D Billman; Charlie K Blake; Shannon Blake; Louis Bliard; Andrea Bonisoli-Alquati; Timothée Bonnet; Camille Nina Marion Bordes; Aneesh P H Bose; Thomas Botterill-James; Melissa Anna Boyd; Sarah A Boyle; Tom Bradfer-Lawrence; Jennifer Bradham; Jack A Brand; Martin I Brengdahl; Martin Bulla; Luc Bussière; Ettore Camerlenghi; Sara E Campbell; Leonardo L F Campos; Anthony Caravaggi; Pedro Cardoso; Charles J W Carroll; Therese A Catanach; Xuan Chen; Heung Ying Janet Chik; Emily Sarah Choy; Alec Philip Christie; Angela Chuang; Amanda J Chunco; Bethany L Clark; Andrea Contina; Garth A Covernton; Murray P Cox; Kimberly A Cressman; Marco Crotti; Connor Davidson Crouch; Pietro B D’amelio; Alexandra Allison de Sousa; Timm Fabian Döbert; Ralph Dobler; Adam J Dobson; Tim S Doherty; Szymon Marian Drobniak; Alexandra Grace Duffy; Alison B Duncan; Robert P Dunn; Jamie Dunning; Trishna Dutta; Luke Eberhart-Hertel; Jared Alan Elmore; Mahmoud Medhat Elsherif; Holly M English; David C Ensminger; Ulrich Rainer Ernst; Stephen M Ferguson; Esteban Fernandez-Juricic; Thalita Ferreira-Arruda; John Fieberg; Elizabeth A Finch; Evan A Fiorenza; David N Fisher; Amélie Fontaine; Wolfgang Forstmeier; Yoan Fourcade; Graham S Frank; Cathryn A Freund; Eduardo Fuentes-Lillo; Sara L Gandy; Dustin G Gannon; Ana I García-Cervigón; Alexis C Garretson; Xuezhen Ge; William L Geary; Charly Géron; Marc Gilles; Antje Girndt; Daniel Gliksman; Harrison B Goldspiel; Dylan G E Gomes; Megan Kate Good; Sarah C Goslee; J. Stephen Gosnell; Eliza M Grames; Paolo Gratton; Nicholas M Grebe; Skye M Greenler; Maaike Griffioen; Daniel M Griffith; Frances J Griffith; Jake J Grossman; Ali Güncan; Stef Haesen; James G Hagan; Heather A Hager; Jonathan Philo Harris; Natasha Dean Harrison; Sarah Syedia Hasnain; Justin Chase Havird; Andrew J Heaton; María Laura Herrera-Chaustre; Tanner J Howard; Bin-Yan Hsu; Fabiola Iannarilli; Esperanza C Iranzo; Erik N K Iverson; Saheed Olaide Jimoh; Douglas H Johnson; Martin Johnsson; Jesse Jorna; Tommaso Jucker; Martin Jung; Ineta Kačergytė; Oliver Kaltz; Alison Ke; Clint D Kelly; Katharine Keogan; Friedrich Wolfgang Keppeler; Alexander K Killion; Dongmin Kim; David P Kochan; Peter Korsten; Shan Kothari; Jonas Kuppler; Jillian M Kusch; Malgorzata Lagisz; Kristen Marianne Lalla; Daniel J Larkin; Courtney L Larson; Katherine S Lauck; M. Elise Lauterbur; Alan Law; Don-Jean Léandri-Breton; Jonas J Lembrechts; Kiara L’herpiniere; Eva J P Lievens; Daniela Oliveira de Lima; Shane Lindsay; Martin Luquet; Ross Macleod; Kirsty H Macphie; Kit Magellan; Magdalena M Mair; Lisa E Malm; Stefano Mammola; Caitlin P Mandeville; Michael Manhart; Laura Milena Manrique-Garzon; Elina Mäntylä; Philippe Marchand; Benjamin Michael Marshall; Charles A Martin; Dominic Andreas Martin; Jake Mitchell Martin; April Robin Martinig; Erin S Mccallum; Mark Mccauley; Sabrina M Mcnew; Scott J Meiners; Thomas Merkling; Marcus Michelangeli; Maria Moiron; Bruno Moreira; Jennifer Mortensen; Benjamin Mos; Taofeek Olatunbosun Muraina; Penelope Wrenn Murphy; Luca Nelli; Petri Niemelä; Josh Nightingale; Gustav Nilsonne; Sergio Nolazco; Sabine S Nooten; Jessie Lanterman Novotny; Agnes Birgitta Olin; Chris L Organ; Kate L Ostevik; Facundo Xavier Palacio; Matthieu Paquet; Darren James Parker; David J Pascall; Valerie J Pasquarella; John Harold Paterson; Ana Payo-Payo; Karen Marie Pedersen; Grégoire Perez; Kayla I Perry; Patrice Pottier; Michael J Proulx; Raphaël Proulx; Jessica L Pruett; Veronarindra Ramananjato; Finaritra Tolotra Randimbiarison; Onja H Razafindratsima; Diana J Rennison; Federico Riva; Sepand Riyahi; Michael James Roast; Felipe Pereira Rocha; Dominique G Roche; Cristian Román-Palacios; Michael S Rosenberg; Jessica Ross; Freya E Rowland; Deusdedith Rugemalila; Avery L Russell; Suvi Ruuskanen; Patrick Saccone; Asaf Sadeh; Stephen M Salazar; Kris Sales; Pablo Salmón; Alfredo Sánchez-Tójar; Leticia Pereira Santos; Francesca Santostefano; Hayden T Schilling; Marcus Schmidt; Tim Schmoll; Adam C Schneider; Allie E Schrock; Julia Schroeder; Nicolas Schtickzelle; Nick L Schultz; Drew A Scott; Michael Peter Scroggie; Julie Teresa Shapiro; Nitika Sharma; Caroline L Shearer; Diego Simón; Michael I Sitvarin; Fabrício Luiz Skupien; Heather Lea Slinn; Grania Polly Smith; Jeremy A Smith; Rahel Sollmann; Kaitlin Stack Whitney; Shannon Michael Still; Erica F Stuber; Guy F Sutton; Ben Swallow; Conor Claverie Taff; Elina Takola; Andrew J Tanentzap; Rocío Tarjuelo; Richard J Telford; Christopher J Thawley; Hugo Thierry; Jacqueline Thomson; Svenja Tidau; Emily M Tompkins; Claire Marie Tortorelli; Andrew Trlica; Biz R Turnell; Lara Urban; Stijn van de Vondel; Jessica Eva Megan van der Wal; Jens van Eeckhoven; Francis van Oordt; K. Michelle Vanderwel; Mark C Vanderwel; Karen J Vanderwolf; Juliana Vélez; Diana Carolina Vergara-Florez; Brian C Verrelli; Marcus Vinícius Vieira; Nora Villamil; Valerio Vitali; Julien Vollering; Jeffrey Walker; Xanthe J Walker; Jonathan A Walter; Pawel Waryszak; Ryan J Weaver; Ronja E M Wedegärtner; Daniel L Weller; Shannon Whelan; Rachel Louise White; David William Wolfson; Andrew Wood; Scott W Yanco; Jian D L Yen; Casey Youngflesh; Giacomo Zilio; Cédric Zimmer; Gregory Mark Zimmerman; Rachel A Zitomer

doi:10.1186/s12915-024-02101-x

Retour

Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Article de revue

Open Access

Avec comité de lecture

Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Elliot Gould, Hannah S Fraser, Timothy H Parker, Shinichi Nakagawa, Simon C Griffith, Peter A Vesk, Fiona Fidler, Daniel G Hamilton, Robin N Abbey-Lee, Jessica K Abbott, …

BMC Biology, Vol.23(1)

06/02/2025

DOI: https://doi.org/10.1186/s12915-024-02101-x

Résumé

Analytical heterogeneity

Metascience

Many-analyst

Replication crisis

Reproducibility

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.

Fichiers et liens (2)

url

Find in HALAfficher

url

https://doi.org/10.1186/s12915-024-02101-xAfficher

Published (Version of record) Ouvrir

Indicateurs

1 Consultations de la notice

See more details

Détails

Titre: Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology
Créateurs - sans rôle: Elliot Gould - The University of Melbourne
Hannah S Fraser - The University of Melbourne
Timothy H Parker - Whitman College
Shinichi Nakagawa - UNSW Sydney
Simon C Griffith - Macquarie University
Peter A Vesk - The University of Melbourne
Fiona Fidler - The University of Melbourne
Daniel G Hamilton - Monash University
Robin N Abbey-Lee
Jessica K Abbott - Lund University
Luis A Aguirre - University of Massachusetts Amherst
Carles Alcaraz
Irith Aloni - Ben-Gurion University of the Negev
Drew Altschul - University of Edinburgh
Kunal Arekar - Indian Institute of Science Bangalore
Jeff W Atkins - Southern Research Station
Joe Atkinson - Aarhus University
Christopher M Baker - The University of Melbourne
Meghan Barrett - University of Indianapolis
Kristian Bell - Deakin University
Suleiman Kehinde Bello - King Abdulaziz University
Iván Beltrán - Macquarie University
Bernd J Berauer - University of Hohenheim
Michael Grant Bertram - Swedish University of Agricultural Sciences
Peter D Billman - University of Connecticut
Charlie K Blake - Southern Illinois University Edwardsville
Shannon Blake - University of Guelph
Louis Bliard - University of Zurich
Andrea Bonisoli-Alquati - California State Polytechnic University
Timothée Bonnet - Centre d'Etudes Biologiques de Chizé
Camille Nina Marion Bordes
Aneesh P H Bose - Swedish University of Agricultural Sciences
Thomas Botterill-James - University of Tasmania
Melissa Anna Boyd - Mammoth Hospital
Sarah A Boyle - Rhodes College
Tom Bradfer-Lawrence - Royal Society for the Protection of Birds
Jennifer Bradham - Wofford College
Jack A Brand - Swedish University of Agricultural Sciences
Martin I Brengdahl - Linköping University
Martin Bulla - Czech University of Life Sciences Prague
Luc Bussière - University of Gothenburg
Ettore Camerlenghi - Monash University
Sara E Campbell - University of Tennessee at Knoxville
Leonardo L F Campos
Anthony Caravaggi - University of South Wales
Pedro Cardoso - University of Lisbon
Charles J W Carroll - Colorado State University
Therese A Catanach - Drexel University
Xuan Chen - Salisbury University
Heung Ying Janet Chik - University of Groningen
Emily Sarah Choy - McMaster University
Alec Philip Christie
Angela Chuang - University of Florida
Amanda J Chunco - Elon University
Bethany L Clark - BirdLife international
Andrea Contina - The University of Texas Rio Grande Valley
Garth A Covernton - University of Toronto
Murray P Cox - University of Auckland
Kimberly A Cressman
Marco Crotti - University of Glasgow
Connor Davidson Crouch - Northern Arizona University
Pietro B D’amelio
Alexandra Allison de Sousa - Bath Spa University
Timm Fabian Döbert - University of Alberta
Ralph Dobler - Technische Universität Dresden
Adam J Dobson - University of Glasgow
Tim S Doherty - The University of Sydney
Szymon Marian Drobniak - Jagiellonian University
Alexandra Grace Duffy - Brigham Young University
Alison B Duncan - Université de Montpellier, Institut des Sciences de l'Evolution - Montpellier - ISEM
Robert P Dunn - University of South Carolina
Jamie Dunning - Imperial College London
Trishna Dutta - European Forest Institute
Luke Eberhart-Hertel
Jared Alan Elmore - Clemson University
Mahmoud Medhat Elsherif - University of Birmingham
Holly M English - University College Dublin
David C Ensminger - San Jose State University
Ulrich Rainer Ernst
Stephen M Ferguson - St. Norbert College
Esteban Fernandez-Juricic - State Street
Thalita Ferreira-Arruda - University of Göttingen
John Fieberg - University of Minnesota
Elizabeth A Finch - CAB International
Evan A Fiorenza - University of California, Irvine
David N Fisher - University of Aberdeen
Amélie Fontaine - McGill University
Wolfgang Forstmeier - Max Planck Institute for Ornithology
Yoan Fourcade - Institut d'écologie et des sciences de l'environnement de Paris
Graham S Frank - Oregon State University
Cathryn A Freund - Wake Forest University
Eduardo Fuentes-Lillo - Institute of Ecology and Biodiversity
Sara L Gandy - University of Glasgow
Dustin G Gannon - Oregon State University
Ana I García-Cervigón - Universidad Rey Juan Carlos
Alexis C Garretson - Tufts University
Xuezhen Ge - University of Guelph
William L Geary - Deakin University
Charly Géron - Université Rennes 2
Marc Gilles - Bielefeld University
Antje Girndt
Daniel Gliksman
Harrison B Goldspiel
Dylan G E Gomes
Megan Kate Good
Sarah C Goslee
J. Stephen Gosnell
Eliza M Grames
Paolo Gratton
Nicholas M Grebe
Skye M Greenler
Maaike Griffioen
Daniel M Griffith
Frances J Griffith
Jake J Grossman
Ali Güncan
Stef Haesen
James G Hagan
Heather A Hager
Jonathan Philo Harris
Natasha Dean Harrison
Sarah Syedia Hasnain
Justin Chase Havird
Andrew J Heaton
María Laura Herrera-Chaustre
Tanner J Howard
Bin-Yan Hsu
Fabiola Iannarilli
Esperanza C Iranzo
Erik N K Iverson
Saheed Olaide Jimoh
Douglas H Johnson
Martin Johnsson
Jesse Jorna
Tommaso Jucker
Martin Jung
Ineta Kačergytė
Oliver Kaltz - Université de Montpellier, Institut des Sciences de l'Evolution - Montpellier - ISEM
Alison Ke
Clint D Kelly
Katharine Keogan
Friedrich Wolfgang Keppeler
Alexander K Killion
Dongmin Kim
David P Kochan
Peter Korsten
Shan Kothari
Jonas Kuppler
Jillian M Kusch
Malgorzata Lagisz
Kristen Marianne Lalla
Daniel J Larkin
Courtney L Larson
Katherine S Lauck
M. Elise Lauterbur
Alan Law
Don-Jean Léandri-Breton
Jonas J Lembrechts
Kiara L’herpiniere
Eva J P Lievens
Daniela Oliveira de Lima
Shane Lindsay
Martin Luquet - Ecologie Comportementale et Biologie des Populations de Poissons
Ross Macleod
Kirsty H Macphie
Kit Magellan
Magdalena M Mair
Lisa E Malm
Stefano Mammola
Caitlin P Mandeville
Michael Manhart
Laura Milena Manrique-Garzon
Elina Mäntylä
Philippe Marchand
Benjamin Michael Marshall
Charles A Martin
Dominic Andreas Martin
Jake Mitchell Martin
April Robin Martinig
Erin S Mccallum
Mark Mccauley
Sabrina M Mcnew
Scott J Meiners
Thomas Merkling - Institut Lorrain du Coeur et des Vaisseaux Louis Mathieu [Nancy]
Marcus Michelangeli
Maria Moiron
Bruno Moreira
Jennifer Mortensen
Benjamin Mos
Taofeek Olatunbosun Muraina
Penelope Wrenn Murphy
Luca Nelli
Petri Niemelä
Josh Nightingale
Gustav Nilsonne
Sergio Nolazco
Sabine S Nooten
Jessie Lanterman Novotny
Agnes Birgitta Olin
Chris L Organ
Kate L Ostevik
Facundo Xavier Palacio
Matthieu Paquet
Darren James Parker
David J Pascall
Valerie J Pasquarella
John Harold Paterson
Ana Payo-Payo
Karen Marie Pedersen
Grégoire Perez - Animal, Santé, Territoires, Risques et Ecosystèmes
Kayla I Perry
Patrice Pottier
Michael J Proulx
Raphaël Proulx
Jessica L Pruett
Veronarindra Ramananjato
Finaritra Tolotra Randimbiarison
Onja H Razafindratsima
Diana J Rennison
Federico Riva
Sepand Riyahi
Michael James Roast
Felipe Pereira Rocha
Dominique G Roche
Cristian Román-Palacios
Michael S Rosenberg
Jessica Ross
Freya E Rowland
Deusdedith Rugemalila
Avery L Russell
Suvi Ruuskanen
Patrick Saccone
Asaf Sadeh
Stephen M Salazar
Kris Sales
Pablo Salmón
Alfredo Sánchez-Tójar
Leticia Pereira Santos
Francesca Santostefano
Hayden T Schilling
Marcus Schmidt
Tim Schmoll
Adam C Schneider
Allie E Schrock
Julia Schroeder
Nicolas Schtickzelle
Nick L Schultz
Drew A Scott
Michael Peter Scroggie
Julie Teresa Shapiro - Unité Epidémiologie et appui à la surveillance
Nitika Sharma
Caroline L Shearer
Diego Simón
Michael I Sitvarin
Fabrício Luiz Skupien
Heather Lea Slinn
Grania Polly Smith
Jeremy A Smith
Rahel Sollmann
Kaitlin Stack Whitney
Shannon Michael Still
Erica F Stuber
Guy F Sutton
Ben Swallow
Conor Claverie Taff
Elina Takola
Andrew J Tanentzap
Rocío Tarjuelo
Richard J Telford
Christopher J Thawley
Hugo Thierry
Jacqueline Thomson
Svenja Tidau
Emily M Tompkins
Claire Marie Tortorelli
Andrew Trlica
Biz R Turnell
Lara Urban
Stijn van de Vondel
Jessica Eva Megan van der Wal
Jens van Eeckhoven
Francis van Oordt
K. Michelle Vanderwel
Mark C Vanderwel
Karen J Vanderwolf
Juliana Vélez
Diana Carolina Vergara-Florez
Brian C Verrelli
Marcus Vinícius Vieira
Nora Villamil
Valerio Vitali
Julien Vollering
Jeffrey Walker
Xanthe J Walker
Jonathan A Walter
Pawel Waryszak
Ryan J Weaver
Ronja E M Wedegärtner
Daniel L Weller
Shannon Whelan
Rachel Louise White
David William Wolfson
Andrew Wood
Scott W Yanco
Jian D L Yen
Casey Youngflesh
Giacomo Zilio - Université de Montpellier, Institut des Sciences de l'Evolution - Montpellier - ISEM
Cédric Zimmer - Laboratoire d’Ethologie Expérimentale et Comparée
Gregory Mark Zimmerman
Rachel A Zitomer
Détails de publication: BMC Biology, Vol.23(1)
Note de subvention: EG’s contributions were supported by an Australian Government Research Training Program Scholarship, AIMOS top-up scholarship (2022) and Melbourne Centre of Data Science Doctoral Academy Fellowship (2021). FF’s contributions were supported by ARC Future Fellowship FT150100297.
Identifiants: 99153239109311
Unité académique: Institut des Sciences de l'Evolution - Montpellier - ISEM
Langue: English
Type de ressource: Journal article
Champs locaux: hal-04943340