-
Tytuł:
-
Wisdom of the expert crowd prediction of response for 3 neurology randomized trials.
-
Autorzy:
-
Atanasov P; From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
Diamantaras A; From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
MacPherson A; From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
Vinarov E; From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
Benjamin DM; From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
Shrier I; From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
Paul F; From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
Dirnagl U; From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
Kimmelman J; From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany. .
-
Źródło:
-
Neurology [Neurology] 2020 Aug 04; Vol. 95 (5), pp. e488-e498. Date of Electronic Publication: 2020 Jun 16.
-
Typ publikacji:
-
Journal Article; Research Support, Non-U.S. Gov't
-
Język:
-
English
-
Imprint Name(s):
-
Publication: Hagerstown, MD : Lippincott Williams & Wilkins
Original Publication: Minneapolis.
-
MeSH Terms:
-
Expert Testimony*
Forecasting*
Neurology*
Randomized Controlled Trials as Topic*
Treatment Outcome*
Humans
-
References:
-
N Engl J Med. 1987 Jul 16;317(3):141-5. (PMID: 3600702)
BMC Public Health. 2015 Aug 11;15:766. (PMID: 26259612)
Lancet Neurol. 2017 Mar;16(3):217-226. (PMID: 28229893)
J Clin Epidemiol. 1997 Jan;50(1):21-9. (PMID: 9048687)
JAMA Dermatol. 2015 Dec 01;151(12):1346-1353. (PMID: 26501400)
JAMA. 2009 Feb 25;301(8):868-9. (PMID: 19244197)
JAMA. 2009 Feb 25;301(8):831-41. (PMID: 19244190)
Neurology. 2016 Jan 12;86(2):126-33. (PMID: 26674335)
J Clin Oncol. 2011 Jan 10;29(2):186-91. (PMID: 21149653)
Med Decis Making. 2016 May;36(4):536-40. (PMID: 25878196)
Neurology. 2010 Apr 6;74(14):1096-101. (PMID: 20368630)
Lancet Neurol. 2018 Aug;17(8):681-688. (PMID: 29934198)
Neurology. 2002 Oct 8;59(7):977-82. (PMID: 12374137)
Proc Natl Acad Sci U S A. 2016 Aug 2;113(31):8777-82. (PMID: 27432950)
PLoS Biol. 2017 Jun 29;15(6):e2002212. (PMID: 28662052)
Resuscitation. 1992 Jun-Jul;23(3):199-206. (PMID: 1321479)
Int J Med Inform. 2016 Aug;92:35-43. (PMID: 27318069)
Stroke. 2012 Mar;43(3):702-7. (PMID: 22223238)
Clin Infect Dis. 2007 Jan 15;44(2):272-9. (PMID: 17173231)
BMJ. 2000 Feb 19;320(7233):469-72. (PMID: 10678857)
-
Grant Information:
-
EOG 201303 Canada CIHR
-
Entry Date(s):
-
Date Created: 20200618 Date Completed: 20201009 Latest Revision: 20221104
-
Update Code:
-
20240105
-
PubMed Central ID:
-
PMC7455341
-
DOI:
-
10.1212/WNL.0000000000009819
-
PMID:
-
32546652
-
Objective: To explore the accuracy of combined neurology expert forecasts in predicting primary endpoints for trials.
Methods: We identified one major randomized trial each in stroke, multiple sclerosis (MS), and amyotrophic lateral sclerosis (ALS) that was closing within 6 months. After recruiting a sample of neurology experts for each disease, we elicited forecasts for the primary endpoint outcomes in the trial placebo and treatment arms. Our main outcome was the accuracy of averaged predictions, measured using ordered Brier scores. Scores were compared against an algorithm that offered noncommittal predictions.
Results: Seventy-one neurology experts participated. Combined forecasts of experts were less accurate than a noncommittal prediction algorithm for the stroke trial (pooled Brier score = 0.340, 95% subjective probability interval [sPI] 0.340 to 0.340 vs 0.185 for the uninformed prediction), and approximately as accurate for the MS study (pooled Brier score = 0.107, 95% confidence interval [CI] 0.081 to 0.133 vs 0.098 for the noncommittal prediction) and the ALS study (pooled Brier score = 0.090, 95% CI 0.081 to 0.185 vs 0.090). The 95% sPIs of individual predictions contained actual trial outcomes among 44% of experts. Only 18% showed prediction skill exceeding the noncommittal prediction. Independent experts and coinvestigators achieved similar levels of accuracy.
Conclusion: In this first-of-kind exploratory study, averaged expert judgments rarely outperformed noncommittal forecasts. However, experts at least anticipated the possibility of effects observed in trials. Our findings, if replicated in different trial samples, caution against the reliance on simple approaches for combining expert opinion in making research and policy decisions.
(© 2020 American Academy of Neurology.)
