Knowledge production in higher education institutions: A policy analysis on how peer review policies caused reviews to change

<a href="https://embed.stanford.edu/iframe/?url=https%3A%2F%2Fpurl.stanford.edu%2Fyj725xj6700" class="su-underline">Show Content</a>

Abstract/Contents

Abstract: A promising approach to estimate the causal effects of peer review policies is to analyze data from publication venues that shift policies from single-blind to double-blind from one year to the next. However, in these settings the content of the manuscript is a confounding variable---each year has a different distribution of scientific content which may naturally affect the distribution of reviewer scores. To address this textual confounding, we extend variable ratio nearest neighbor matching to incorporate text embeddings. We compare this matching method to a widely-used causal method of stratified propensity score matching and a baseline of randomly selected matches. For our case study of the ICLR conference shifting from single- to double-blind review from 2017 to 2018, we find human judges prefer manuscript matches from our method in 71% of cases. Our results show that the double-blind review policy causes a negative change in the review score, politeness score, and sentiment score. This work hopes to contribute to the progression of text-based causal estimation methods and a case study of knowledge production in higher education institutions.

Type of resource	text
Publication date	June 9, 2023

Author	Zhang, Raymond
Advisor	McFarland, Daniel

Subject	Causation
Subject	Natural language processing
Subject	Natural language understanding
Subject	Science of science
Subject	Peer review
Genre	Text
Genre	Capstone
Genre	Student project report

Location	Location https://github.com/raymondEDS/VRM-E
DOI	https://doi.org/10.25740/yj725xj6700
Location	https://purl.stanford.edu/yj725xj6700

Use and reproduction: User agrees that, where applicable, content will not be used to identify or to otherwise infringe the privacy or confidentiality rights of individuals. Content distributed via the Stanford Digital Repository may be subject to additional license and use restrictions applied by the depositor.
License: This work is licensed under a Creative Commons Attribution Non Commercial 4.0 International license (CC BY-NC).

Preferred citation: Zhang, R. (2023). Knowledge production in higher education institutions: A policy analysis on how peer review policies caused reviews to change. Stanford Digital Repository. Available at https://purl.stanford.edu/yj725xj6700. https://doi.org/10.25740/yj725xj6700.

Education Data Science (EDS) Capstone Projects, Graduate School of Education

Loading usage metrics...