| Panel split | Frequency | % |
|---|---|---|
| 5–0 | 3 | 23.1 |
| 4–1 | 3 | 23.1 |
| 3–2 | 7 | 53.8 |
Are referees judged harshly with respect to VAR?
football
statistics
Premier League
Examining the futility of the Premier League’s Key Match Incident Panel using Bayesian probability.
There are three different camps when it comes to VAR in football. Those, like me, who have been against VAR from day one: I think it ruins the excitement of goals and creates unrealistic expectations of forensic-level accuracy over every decision. There are also those who think VAR is fantastic and are very happy with it. I mean, I’ve never met those people, but they must exist. And the third camp consists of those who think VAR is a good idea, if only those pesky humans operating it weren’t so incompetent!
The problem with that last group is, I’m not sure people are being realistic in terms of how easy it is to get refereeing decisions right.
This BBC article describes the increase in VAR errors in the first half of the current Premier League season, and lists the 13 VAR “mistakes”, 11 of which were missed interventions and two of which were incorrect interventions.
After giving details of each incident, the article also shows a running total of which teams have suffered from or gained from these errors, just to pour a bit more petrol on the bonfire of those who think the world is conspiring against their team. Football supporters have never been renowned for their objectivity.
The mistakes are identified by the Key Match Incidents (KMI) Panel which meets every week to adjudicate on all key decisions in Premier League matches. The panel consists of one representative each from the Premier League and Professional Game Match Officials Limited (PGMOL), as well as three from a rotating pool of former players and managers including the likes of Karen Carney, Rob Green, Terry Burton, Jonathan Walters and Steven Reid.
We will gloss over the suitability of those individuals for the panel - judging by the general standard of punditry and co-commentary from ex-players on TV, the bar is not high - but even if we respect the judgement of the panel, what are the statistics behind these mistakes actually showing us?
According to the BBC article, here is the breakdown of the panel’s votes on the 13 VAR errors:
More than half the ‘mistakes’ were only decided by a 3-2 panel verdict.
Evidently, of the 13 decisions branded as mistakes, only three of them were unanimous 5-0 decisions by the KMI Panel. Three of them had one dissenter in a 4-1 decision and in more than half of the so-called mistakes, the panel was split 3-2! So can we really say that the referees got it wrong? What if the panel got it wrong?
A Bayesian approach
The language surrounding refereeing decisions in football can be quite binary and hyperbolic. You will often hear “that’s a stonewall penalty”, or “never a pen in a million years”; sometimes about the same decision!
The fact that the panel of experts failed to unanimously agree on most of the cases shows that reality is not so clear-cut. It is reasonable to consider each refereeing decision not as a simple case of right or wrong, but instead as having a probability somewhere between 0 and 1 of being incorrect.
Under that assumption, we can model the KMI Panel’s voting split as a beta-binomial distribution, where the underlying probability of the decision being incorrect follows a beta distribution, and then our five expert panelists are treated as independent Bernoulli draws and may vote correct or incorrect based on that underlying probability.
Unfortunately we don’t have the full data relating to the KMI Panel’s votes on every case, only on the mistakes, so we can’t derive the full distribution. But what we can do is use a few different example priors, and show what our posterior probability would be of the referee’s decision being incorrect after a 3-2, 4-1, or 5-0 vote.
The obvious starting point for the prior is the uniform beta (1,1). This is a flat line: we simply don’t know anything about the probability of the referee’s decision being correct or incorrect. But this also implies a 50% overall probability of the referee’s decision being wrong, which is too high; we know from the article that there were, at minimum, 49 VAR interventions, plus other cases the panel considered while only finding 13 to be mistakes.
The second and third beta distributions I have chosen assume a mean of 0.1 probability that the referee’s decision was incorrect. They vary in shape: the second distribution is a downward curve, while the third distribution is U-shaped, reflecting a larger frequency near 0 and 1: your stonewall and never-in-a-million-years decisions.
Based on these three prior distributions we can calculate the posterior probability of the ref’s decision being wrong, conditional on the panel’s votes.
| 5-0 | 4-1 | 3-2 | |
|---|---|---|---|
| Beta(0.05, 0.45) | 0.92 | 0.74 | 0.55 |
| Beta(0.3, 2.7) | 0.66 | 0.54 | 0.41 |
| Beta(1.0, 1.0) | 0.86 | 0.71 | 0.57 |
Mean posterior probability of decision being incorrect, conditional on observed panel decision.
So, depending on your prior, the maximum estimate of the probability of an incorrect decision based on the panel’s vote would be 92% for 5-0, 74% for 4-1, and 57% for 3-2: not far off a coin flip!
This sort of thing highlights a wider problem with statistics which is that there are different ways to spin the exact same result. The BBC have gone with “VAR mistakes are on the rise!” whereas I look at the panel’s voting splits and think this just shows you how difficult a referee’s job is: even a panel of experts, with all the time in the world, cannot agree what is the correct decision.
So what is a better approach? Stop reporting 3-2 decisions as errors. At minimum, I think they should require at least a 4-1 vote for a decision to be declared incorrect, and preferably 5-0. This is in line with the “clear and obvious error” spirit that everyone seems to be in favour of (until there is a decision they don’t like).
A similar system exists in a jury, where all 12 members have to agree to reach a guilty verdict. Perhaps we should have a VAR jury in the Premier League? Five different refs sat in different rooms, and a 4-1 majority required to overturn the on-field decision.
Nonetheless, I feel like the accuracy of VAR is a red herring. The overall accuracy of decisions has improved compared to pre-VAR, and yet nobody seems happier. There will continue to be incremental improvements, but that will just move us to a new frontier of what everyone will complain about; we can already look forward to VAR reviews of corners. Exciting!
Those who anticipated VAR ushering in a new era without weekly officiating controversies were sadly mistaken. We did lose a lot of the spontaneous elation when a goal is scored - the new normal is that your team has probably scored, but you are worried about that pass in the buildup where the winger had maybe strayed half a yard offside. Scoring is now more like getting a penalty: there’s a good chance we are getting a goal here. Was it worth the trade? Not for me.
© 2025 John Knight. All rights reserved.