The Effects of Sample Size on Significance Testing

Placeholder Show Content


There are two ways in which a researcher might make incorrect conclusions based on an empirical study's statistical test: false negatives and false positives. Basic statistics highlight the dangers of small samples on false negatives, which is vastly known. Further, statistical significance tests produce a p-value associated with false positives. Thus, while sample size is related to the probability of a false-negative conclusion, it should be unrelated to the probability of a false-positive conclusion. Despite the logic underlying statistical testing, there are rising concerns about the increased chances of producing a false-positive error as the sample size increases. Since scientists have yet to test this relation in depth, this paper leverages data from five independent studies testing five different effects (i.e., 25 samples of ~1500 participants) to examine the effects of sample size on false positives. This paper systematically tests an implausible hypothesis — whether random assignment into experimental and control conditions can predict a participant's demographics — using simulated samples of 50 to 5000. Initial analyses found that large samples were more likely to yield false-positive results than small samples. Subsequent analyses suggest that weak effects of differential attrition, which may be expected in any experiment, may only be detectable in large sample studies. Therefore, large-sampled studies may find experimental effects that are artifacts caused by differential attrition.


Type of resource text
Date created February 28, 2022
Date modified December 5, 2022
Publication date February 28, 2022


Author Jordan, Diana
Thesis advisor Krosnick, Jon


Subject sample size
Subject Statistical hypothesis testing
Subject false negatives
Subject false positives
Subject type II error
Subject type I error
Subject attrition
Subject power
Genre Text
Genre Thesis

Bibliographic information

Access conditions

Use and reproduction
User agrees that, where applicable, content will not be used to identify or to otherwise infringe the privacy or confidentiality rights of individuals. Content distributed via the Stanford Digital Repository may be subject to additional license and use restrictions applied by the depositor.
This work is licensed under a Creative Commons Attribution Non Commercial No Derivatives 4.0 International license (CC BY-NC-ND).

Preferred citation

Preferred citation
Jordan, D. (2022). The Effects of Sample Size on Significance Testing. Stanford Digital Repository. Available at


Undergraduate Honors Theses, Department of Communication, Stanford University

View other items in this collection in SearchWorks

Contact information

Also listed in

Loading usage metrics...