Effects of Training Novice Raters in Creativity
Advisor Information
Roni Reiter-Palmon
Location
Dr. C.C. and Mabel L. Criss Library
Presentation Type
Poster
Start Date
4-3-2016 10:45 AM
End Date
4-3-2016 12:15 PM
Abstract
One problem that creativity researchers face while studying creativity is how to accurately measure the construct. The construct of creativity is most commonly described as products or ideas that are of both high quality and originality. There are many different levels or degrees to which ideas or products can be of high quality and originality, but the Consensual Assessment Technique (CAT), developed by Teresa Amabile, is widely accepted as the best measure of creativity. The CAT uses expert raters for creativity research because they yield high interrater reliability and are the best judges of quality and originality within their domain. One problem with the CAT method is in finding and retaining experts to conduct the ratings, which can often be time consuming and expensive. This is why many creativity researchers often rely upon novices, or non-expert raters, to analyze and rate the creativity of solutions. The purpose of this study was to determine the training conditions necessary to optimize novice creativity ratings to be more similar and consistent to the ratings of experts. Utilizing electronic methods, we trained novice raters for both quality and originality using multiple rubrics ranging in quality, while also manipulating whether or not their training included practice ratings with feedback or not. Results showed that practice ratings and detailed rubrics increased the accuracy of ratings done by participants. Interestingly, this relationship was true only for the ratings of quality, not originality.
Effects of Training Novice Raters in Creativity
Dr. C.C. and Mabel L. Criss Library
One problem that creativity researchers face while studying creativity is how to accurately measure the construct. The construct of creativity is most commonly described as products or ideas that are of both high quality and originality. There are many different levels or degrees to which ideas or products can be of high quality and originality, but the Consensual Assessment Technique (CAT), developed by Teresa Amabile, is widely accepted as the best measure of creativity. The CAT uses expert raters for creativity research because they yield high interrater reliability and are the best judges of quality and originality within their domain. One problem with the CAT method is in finding and retaining experts to conduct the ratings, which can often be time consuming and expensive. This is why many creativity researchers often rely upon novices, or non-expert raters, to analyze and rate the creativity of solutions. The purpose of this study was to determine the training conditions necessary to optimize novice creativity ratings to be more similar and consistent to the ratings of experts. Utilizing electronic methods, we trained novice raters for both quality and originality using multiple rubrics ranging in quality, while also manipulating whether or not their training included practice ratings with feedback or not. Results showed that practice ratings and detailed rubrics increased the accuracy of ratings done by participants. Interestingly, this relationship was true only for the ratings of quality, not originality.