Can Managers Rate Employee Skills Effectively?

Most bosses think they know their employees better than anyone else.

After all, we spend 90,000 hours of our lives at work. And it's the team leaders who are in the trenches every day, training employees on important processes and coaching them on their way to achieving business goals.

But knowing someone well, and rating them well are two different things.

The impact of "rater effects"

That's according to a study conducted by Michael Mount, Steven Scullen, and Maynard Goff published in the Journal of Applied Psychology in 2000. Mount and his team researched 4,492 managers who were rated on performance by two bosses, two peers, and two subordinates. They found that 62% of the variance in the ratings could be accounted for by individual raters’ perception (a.k.a. “idiosyncratic rater effects”). Actual performance accounted for only 21% of the variance.

As researchers put it, "Ideally, the rating variance associated with the performance of the ratee would be large relative to the variance associated with biases of the rater. In other words, what is being rated should account for more variance than who does the rating. Our results show that this is not generally true."

As it turns out, the way you rate your employees is more a reflection of your own thoughts and beliefs about work performance, and less about the actual performance of the person you're rating.

But so what does this mean for performance ratings?

A real-world solution

Deloitte came up with a simple solution. After reviewing the research and conducting their own internal survey, the Big Four firm found that although its previous review system was considered fair and effective by most employees, it just wasn't moving the needle on business objectives. Not only that, they found that over 2 million hours a year were spent on the review process.

In an effort to reduce the time burden and outsmart "rater effects", Deloitte now asks team leaders what they would do with their employees, rather than asking them to rate employees on specific skills. For example, the first of their four review questions is, "Given what I know of this person’s performance, and if it were my money, I would award this person the highest possible compensation increase and bonus." Managers are then asked to agree or disagree using a 5-point scale. This question has the reviewer reflect on their own feelings rather than try to give a score to another person’s skills. We are much better equipped to do the former, rather than the latter.

"Boss ratings" are still the best we've got

Deloitte's radically simple approach helps uncover the manager's core insights, without asking them to ponder abstract terms like 'critical reasoning' and 'strategic thinking skills'. Unfortunately, the study didn't investigate the cause of individual "rater effects", but it's safe to assume that using vague terminology to describe employee skills could leave the door open to a little too much interpretation.

And that's a problem, because as researchers pointed out, "Regardless of whether perspective-related effects are classified as actual performance or bias, our results indicate that boss ratings capture more of the ratee's actual job performance than do ratings from any other perspective."

In other words, managers and team leaders may not be able to rate employee skills effectively, but they still hold the key to getting the most accurate reading possible. Just how accurate that is may have more to do with how you ask, and less to do with what you think.