It’s not you, it’s me. The idiosyncratic rater effect

Published
04 Nov 2019

04 Nov 2019

Picture the scene. It’s time for those annual appraisals we all know and love. As a manager, you’re marshalling everything you need to explore and quantify how your people are doing. You have KPIs, you’re following your organisation’s rating system.

But hang on. Is this person really a ‘5’ as a communicator? How does that compare with other members of the team? Can I be really be sure about the criteria? What’s the difference between, say, a 3 and a 4? Will I be seen as a stickler if I apply the criteria strictly? Or can I cut my team a little slack? 

Suddenly, we’re beginning to have some empathy with those TV talent show judges whose objectivity often seems a little questionable. 

In an ideal world, we’d all like to think that we are pretty reliable narrators of other people’s performance at work: how effective they are as decision makers; how much attention they pay to detail; whether they have leadership potential. After all, isn't it enough for us to observe how someone shows up regularly for us to be able to rate them accurately?

Unreliable narrators

And what would we do if we didn’t rate our people? How else would we differentiate between employees? Decide who gets the big promotion, or who needs a blunt conversation in a small room, with a bunch of tissues at hand? Ratings give us the framework and the confidence to make decisions between our people; they’re the glue that holds together most organisational performance management systems, appraisals and 360° reviews, those competency frameworks. 

When it comes to plotting performance, however, there’s an elephant in the room that can be rather awkward to acknowledge. For many years now, a range of research has revealed that relying on our ability to rate others – whatever the format or context – is not just unreliable, but, according to author and consultant, Marcus Buckingham, “ludicrous”. 

Harmful decisions

In an article for Harvard Business Review, Buckingham challenges the received wisdom that “with enough training and time, people can become reliable raters of other people”. Instead, he gives us the results of three psychological studies published between 1998 and 2010 that shine a light on the phenomenon Buckingham calls the idiosyncratic rater effect

The bottom line is that we’re disturbingly unreliable as raters of other people’s performance. In fact, the studies showed that more than half of the variation in a manager’s ratings could be explained by the “unique rating patterns of the individual doing the rating”. That means that, on average, 61% – by far the larger part – of our ratings of colleagues say more about ourselves than the person we’re rating. It seems that it really is about us, and not them. 

With all of this in mind, those apparently harmless inner monologues we all have about rating our colleagues might seem a bit less harmless, especially if those ratings are used for a whole raft of crucial decisions from the allocation of training budgets to salary increases or promotions.

It’s sobering to think that, when we are asked to rate a colleague on their ‘leadership’ or ‘communication’ skills, the largest influence on our verdict will come from our own idiosyncrasies and biases, what we consider good leadership or clear communication to look like; how experienced or competent we are in these areas. 

So, what can we do about it? 

The fact that we are unreliable raters of others’ abilities is not to say that we can’t be reliable raters of our own experience of others. We can tell colleagues, with a high degree of accuracy, whether they inspired us when gave the presentation, or whether we found what they were saying just now about the trouble in the supply chain a little hard to understand. What we can reliably ask ourselves, then, is questions about our own experiences of others, provided that we do so giving specific examples and in a fairly granular way.  

What we can’t do is say whether Jack or Jasmine is a good or bad leader or communicator in an unequivocally objective way. In a sense, the lesson of the idiosyncratic rater effect is that we simply need to be much humbler about what we can realistically feed back on. The more common lack of humility in giving people hard and fast ‘objective’ ratings is precisely one of the reasons performance management at work can be so demotivating. We might all think carefully about the observation from Danish philosopher, Soren Kierkegaard, that “when you label me, you negate me”. 

But we shouldn’t take it too personally; the inability to rate others accurately happens to the best of us. Just think of all the teeth-gnashing at the many UK publishers who turned down J K Rowling’s first Harry Potter novel. 

Discover our Transformational Learning Programme