replication crisis - 2025 ACF Fall

Question

The study of this problem was popularized after Daryl Bem reported experimental evidence for ESP. For 10 points each:

[10m] What so-called “crisis” in social science is driven by “publish or perish” culture? A John Ioannidis article on this problem claims “Most Published Research Findings Are False.”

ANSWER: replication crisis [or replicability crisis or reproducibility crisis or nonreproducibility crisis]

[10e] A proposed solution to the replication crisis lowers the threshold for this property from “p is less than 0.05” to “p is less than 0.005.” Results with this property’s “statistical” form are unlikely to have happened by chance.

ANSWER: statistical significance [or word forms like statistically significant]

[10h] Due to publication biases, even successful replications typically find lower values of this quantity. Cohen’s d estimator for this quantity equals the difference between two means over the standard deviation.

ANSWER: effect size

<Editors, Social Science>

Back to bonuses

Summary


Northern California	Main	Y	2	15.00	100%	50%	0%
Southern California	Main	Y	7	18.57	100%	71%	14%
Eastern Canada (1)	Main	Y	2	10.00	50%	50%	0%
Eastern Canada (2)	Main	Y	5	12.00	60%	60%	0%
Florida	Main	Y	2	5.00	0%	50%	0%
Great Lakes	Main	Y	10	14.00	90%	40%	10%
Lower Mid-Atlantic	Main	Y	6	16.67	83%	83%	0%
Upper Mid-Atlantic	Main	Y	1	20.00	100%	100%	0%
Midwest	Main	Y	7	12.86	86%	43%	0%
North	Main	Y	2	20.00	100%	50%	50%
Northeast	Main	Y	6	15.00	83%	67%	0%
Pacific	Main	Y	6	16.67	67%	100%	0%
South Central	Main	Y	4	12.50	75%	50%	0%
Southeast	Main	Y	8	10.00	63%	38%	0%
UK (North)	UK	Y	3	10.00	33%	33%	33%
UK (South)	UK	Y	5	16.00	80%	60%	20%
Upstate NY	Main	Y	2	20.00	100%	100%	0%

Data


Berkeley Mighty Miner	Berkeley Boss Bandit	10	10	0	20
Stanford A	Berkeley Little Prince	0	10	0	10
Claremont A	Claremont B	10	10	0	20
Carleton A	Carleton B	10	10	0	20
McGill B	McGill A	0	0	0	0
UCLA A	USC A	0	10	0	10
UCLA D	UCLA B	10	10	0	20
UCSD B	UCLA C	10	10	10	30
USC B	UCLA E	0	10	0	10
UCLA F	UC Irvine	10	10	0	20
UC Santa Barbara	UCSD A	10	10	0	20
Toronto E	McMaster B	10	10	0	20
Queen's A	Toronto F	0	10	0	10
McMaster A	Toronto B	10	10	0	20
Toronto G	Guelph A	0	0	0	0
Waterloo A	Toronto D	10	0	0	10
Florida B	Florida A	10	0	0	10
Florida Tech A	UCF A	0	0	0	0
Michigan A	CWRU A	10	10	0	20
CWRU B	Louisville A	10	10	10	30
CWRU D	Ohio State B	0	10	0	10
Kentucky	Louisville D	0	10	0	10
Michigan State A	Michigan B	10	10	0	20
Louisville B	Michigan C	0	10	0	10
Michigan State B	Ohio State A	0	10	0	10
CWRU C	Ohio State C	10	10	0	20
Ohio State D	Louisville C	0	10	0	10
Pittsburgh	Michigan D	0	0	0	0
NCSU A	UNC A	10	10	0	20
Liberty A	Duke A	0	0	0	0
VCU A	UNC B	10	10	0	20
William & Mary A	Virginia A	10	10	0	20
Virginia Tech A	Virginia Tech B	10	10	0	20
William & Mary C	William & Mary B	10	10	0	20
George Mason A	Maryland A	10	10	0	20
Chicago B	Illinois Urbana	10	10	0	20
Iowa A	Chicago A	0	10	0	10
Purdue B	Iowa B	0	10	0	10
Rose-Hulman A	William Woods University A	0	10	0	10
Purdue D	WashU Grolars	0	0	0	0
Wisconsin A	St. Thomas	10	10	0	20
Wisconsin C	Minnesota	0	10	10	20
Amherst B	Tufts C	10	10	0	20
MIT A	Boston College A	0	10	0	10
Brown A	Amherst A	0	0	0	0
Vermont B	Dartmouth A	10	10	0	20
UConn A	Tufts B	10	10	0	20
Brandeis Mozart	Williams B	10	10	0	20
Illinois Champaign	Illinois Blue	10	10	0	20
Purdue A	Purdue C	10	10	0	20
Alberta A	Washington Steelhead	10	10	0	20
Washington State	Arizona State A	10	0	0	10
British Columbia A	Texas	10	0	0	10
Washington Atlantic	British Columbia B	10	10	0	20
Washington Chum	British Columbia C	10	10	0	20
Washington Pink	Washington Sockeye	10	10	0	20
Kansas State A	Central Oklahoma B	10	10	0	20
Central Oklahoma A	Nebraska B	0	10	0	10
Murray State College A	Oklahoma	0	0	0	0
Nebraska A	Tulsa A	10	10	0	20
Auburn B	Auburn A	10	0	0	10
Chipola College	Alabama A	0	10	0	10
Tennessee Tech	Georgia A	0	0	0	0
Clemson A	Georgia Tech C	10	10	0	20
Georgia Tech D	Montevallo	0	0	0	0
Georgia Tech E	Emory A	0	10	0	10
Vanderbilt A	South Carolina A	0	10	0	10
Georgia Tech A	Tennessee A	10	10	0	20
Durham A	Edinburgh A	0	0	0	0
Edinburgh C	Durham C	10	10	10	30
York	Edinburgh D	0	0	0	0
Southampton	Birmingham	10	0	0	10
Cambridge C	Imperial B	0	10	0	10
Cambridge B	Cardiff	10	10	10	30
Imperial A	Warwick	0	10	0	10
Cambridge A	Oxford	10	10	0	20
Cornell A	ESF	10	10	0	20
RIT A	Cornell C	10	10	0	20