Distributional properties of the statistic of online student evaluations the mean does not mean what you think it means

Authors

Ben O. Smith, University of Nebraska at OmahaFollow
Dustin R. White, University of Nebraska at OmahaFollow
Jamie Frances Wagner, University of Nebraska at OmahaFollow
Patricia C. Kuzyk, Washington State University
Alex Prera, Washington State University

Author ORCID Identifier

Smith - https://orcid.org/0000-0003-1286-0852

White - https://orcid.org/0000-0002-2566-5338

Wagner- https://orcid.org/0000-0003-0527-8282

Document Type

Article

Publication Date

5-10-2023

Publication Title

Studies in Higher Education

Abstract

Student Evaluations of Teaching (SETs) are an integral part of evaluating course outcomes. They are routinely used to evaluate teaching quality for the purposes of reappointment, promotion, and tenure (RPT), annual review, and the rehiring of adjunct faculty and lecturers. These evaluations are often based almost entirely on the mean or proportion of the ordinal overall score with no regard to statistical noise. This study examines the distribution of the statistic (mean or proportion) when SETs are administered online and in-person. Using non-parametric procedures, we show that the size of the 95% confidence interval of the statistic is a function of response rates. Prior to COVID-19, online administration of SETs resulted in significantly more uncertainty than in-person administration because the in-person response rates were higher. Due to a decrease in in-person response rates in the post-COVID vaccine period, both methods result in significant levels of uncertainty of the true statistic value. In classes of fewer than 30 students, the 95% confidence interval of the statistic is wide enough for instructors to be considered for a teaching award in one semester or below average in another semester, while holding teaching quality constant.

Comments

This is an Accepted Manuscript of an article published by Taylor & Francis in [Studies in Higher Education] on [May 10, 2023], available online: https://doi.org/10.1080/03075079.2023.2211079

Recommended Citation

To cite this article: Ben O. Smith, Dustin R. White, Jamie Wagner, Patricia Kuzyk & Alex Prera (2023): Distributional properties of the statistic of online student evaluations the mean does not mean what you think it means, Studies in Higher Education, DOI: https://doi.org/10.1080/03075079.2023.2211079

Creative Commons License

This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.