Computational validity
Abstract/Contents
- Abstract
- As online learning platforms and computerized testing become more common, an increasing amount of data are collected about users. These data include, but are not limited to, response time, keystroke logs, and raw text. The desire to observe these features of the response process reflect an underly- ing interest in the cognitive processes and behaviors respondents engage in while taking tests and navigating learning platforms. Alongside this interest in response processes is an increased desire to use computational methods, artificial intelligence, and machine learning to combine this process data with traditional item response data. This dissertation attempts to advance a notion of computational validity, defined as arguments for the use and interpretation of test scores from evidence collected or analyzed using computational methods. Chapter one looks at item position effects as a threat to comparability across test forms. The presence of position effects is especially threatening to computer adaptive tests. Here a mixture item response theory model that decomposes item and person components of position effects is developed and applied to a large Brazilian college admissions exam to demonstrate its ability to both recover information about position effects in items and automatically produce comparable ability distributions across forms. Chapter two looks at data from a browser-based reading assessment and proposes drift diffusion models as a theoretically grounded way to combine response time and response accuracy data. The individual parameters estimated from the drift diffusion models are combined with scores on the browser-based assessment and used to predict scores on more expensive and time consuming measures of reading ability. This provides a line of evidence justifying the clinical and classroom use of the tool for diagnostic purposes. Chapter three looks at the issue of model selection. To make decisions from a model requires justification that the model is appropriate for the data. Current methods of model selection are well suited to make decisions on which model fits better, but not to communicate about how much better or if the degree of fit is appropriate. This paper uses a tool called the InterModel Vigorish that aims to provide measures of increases in predictive performance on an interpretable scale and extends it to polytomous item response theory models. Simulations are performed to explore what can be learned about data generating processes from this tool and then it is applied to a selection of empirical datasets. Combined, this work aims to highlight ways that modern computational methodologies can be used to improve not only assessments, but the decisions we make from them and our ability to communicate about them.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2023; ©2023 |
| Publication date | 2023; 2023 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Kanopka, Klint |
|---|---|
| Degree supervisor | Ruiz-Primo, Maria Araceli |
| Thesis advisor | Ruiz-Primo, Maria Araceli |
| Thesis advisor | Domingue, Ben |
| Thesis advisor | Yeatman, Jason |
| Degree committee member | Domingue, Ben |
| Degree committee member | Yeatman, Jason |
| Associated with | Stanford University, Graduate School of Education |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Klint Kanopka. |
|---|---|
| Note | Submitted to the Graduate School of Education. |
| Thesis | Thesis Ph.D. Stanford University 2023. |
| Location | https://purl.stanford.edu/tr545td7650 |
Access conditions
- Copyright
- © 2023 by Klint Kanopka
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...