The Effects of Sample Size on Significance Testing
Abstract/Contents
- Abstract
- There are two ways in which a researcher might make incorrect conclusions based on an empirical study's statistical test: false negatives and false positives. Basic statistics highlight the dangers of small samples on false negatives, which is vastly known. Further, statistical significance tests produce a p-value associated with false positives. Thus, while sample size is related to the probability of a false-negative conclusion, it should be unrelated to the probability of a false-positive conclusion. Despite the logic underlying statistical testing, there are rising concerns about the increased chances of producing a false-positive error as the sample size increases. Since scientists have yet to test this relation in depth, this paper leverages data from five independent studies testing five different effects (i.e., 25 samples of ~1500 participants) to examine the effects of sample size on false positives. This paper systematically tests an implausible hypothesis — whether random assignment into experimental and control conditions can predict a participant's demographics — using simulated samples of 50 to 5000. Initial analyses found that large samples were more likely to yield false-positive results than small samples. Subsequent analyses suggest that weak effects of differential attrition, which may be expected in any experiment, may only be detectable in large sample studies. Therefore, large-sampled studies may find experimental effects that are artifacts caused by differential attrition.
Description
| Type of resource | text |
|---|---|
| Date created | February 28, 2022 |
| Date modified | December 5, 2022 |
| Publication date | February 28, 2022 |
Creators/Contributors
| Author | Jordan, Diana |
|---|---|
| Thesis advisor | Krosnick, Jon |
Subjects
| Subject | sample size |
|---|---|
| Subject | Statistical hypothesis testing |
| Subject | false negatives |
| Subject | false positives |
| Subject | type II error |
| Subject | type I error |
| Subject | attrition |
| Subject | power |
| Genre | Text |
| Genre | Thesis |
Bibliographic information
Access conditions
- Use and reproduction
- User agrees that, where applicable, content will not be used to identify or to otherwise infringe the privacy or confidentiality rights of individuals. Content distributed via the Stanford Digital Repository may be subject to additional license and use restrictions applied by the depositor.
- License
- This work is licensed under a Creative Commons Attribution Non Commercial No Derivatives 4.0 International license (CC BY-NC-ND).
Preferred citation
- Preferred citation
- Jordan, D. (2022). The Effects of Sample Size on Significance Testing. Stanford Digital Repository. Available at https://purl.stanford.edu/gc542kg1237
Collection
Undergraduate Honors Theses, Department of Communication, Stanford University
View other items in this collection in SearchWorksContact information
- Contact
- dianajor@stanford.edu
Also listed in
Loading usage metrics...