Should We Always Trust Computers from a Legal Perspective?

The Post Office scandal has highlighted the fact that computer systems are not always right. This assumption has caused hundreds of people to be wrongly convicted. So, legally, what led to this incorrect assumption?

A document produced by a computer is “hearsay” evidence, which means the person relying on it has no direct personal knowledge of it. If a witness said “He told me that Simon said XXXX” then the courts would generally give less weight to XXXX, because it is hearsay evidence.

However, the legal presumption that computers are reliable stems from an older common law principle that “mechanical instruments” should be presumed to be in working order unless proven otherwise. That assumption means that if, for example, a police officer quotes the time on their watch, a defendant cannot force the prosecution to call an expert watchmaker to explain from first principles how watches work.

In 1984, s.69 of the Police and Criminal Evidence Act (PACE) required the prosecution to prove that a computer was working correctly at the relevant time before a document produced by the computer could be admitted in evidence.  This proved cumbersome and was replaced in 1999 by the Youth Justice and Criminal Evidence Act which makes the presumption that mechanical instruments and computers were in order at the material time in the absence of evidence to the contrary.

The purpose of this presumption was to streamline the process and save the time and expense of proving the obvious.  However, it fundamentally changes the burden of proof as now the defendant has to prove that the computer was faulty.  As computers became more complicated, this was impractical and few defendants had the expertise or financial backing necessary to challenge the reliability and integrity of a computer system.

There is also a fundamental difference between computer hardware and software.  It is probably fair to say that hardware is more akin to “mechanical instruments” and rarely goes wrong, but software is a novel concept for the law and we all know that it does go wrong – as indeed was the case for the Post Office Horizon system.  Indeed, we now accept that most software has some bugs in it and is not perfect; consider the ubiquitous concept of version 2.0 and beyond.

Artificial Intelligence (AI) highlights this problem even further.  Machine-learning-based systems are even less like “mechanical instruments”.  They are probabilistic and are often specifically designed not to produce the same result for the same input so we cannot count on them to behave consistently, only to work in line with their projected accuracy.

There have been several cases involving people who have relied on AIs to prepare their cases at court only to discover that the AI has invented cases that do not actually exist!

There is clearly a need to incorporate the normal everyday experience of most of us that computer systems sometimes get things wrong, back into the fundamentals of our legal system.