Rating systems have been relied upon for years to improve performance. Often times a person’s career, pay raise or job is in the balance.
How effective is your performance management rating system?
A study published in the Journal of Applied Psychology sheds a not too flattering light on rating systems and what they really measure.
“The most comprehensive research on what ratings actually measure was conducted by professors Mount, Scullen, and Goff. In their study, 4,492 individuals were rated on a number of different performance dimensions by two bosses, two peers and two subordinates, who combined to produce almost half a million ratings. The researchers then analyzed these ratings and discovered that 54% of the variance in the ratings could be accounted for by “idiosyncratic rater effects”—namely the peculiarities of each individual rater’s perception. Only 21% of the variance in ratings could be explained by the ratee’s actual performance. All of which led the researchers to the following conclusion:
“Although it is implicitly assumed that the ratings measure the performance of the ratee, most of what is being measured by the ratings is the unique rating tendencies of the rater. Thus ratings reveal more about the rater than they do about the ratee.”
Scullen, S., Mount, M., & Goff, M. (2000). Understanding the latent structure of job performance ratings. J Appl Psychol., 85(6), 956–70.
What’s your experience been with rating systems in your organization? Do you feel they are worth the time and effort?