Undoubtedly the TV event of the year so far has been ITV’s Mr Bates vs The Post Office. Up to 10 million viewers were gripped by the story of what has been labelled one of the greatest miscarriages of justice in British history, the prosecution of more than 700 sub-postmasters for theft and fraud they did not commit.
You don’t expect a POS system to be at the heart of such a gripping and harrowing tale. But front and centre of the scandal was the Horizon ePOS and accounting system introduced to Post Office branches in 1999.
The gaping holes in branch accounts that appeared time and time again in the following years weren’t caused by the criminal deceit of hundreds of agents, as the Post Office went on to allege. They were caused by serious failings in the Horizon system.
As well as being the very human story of how ordinary people can have their lives shattered by false accusations, the Post Office scandal is a timely warning of what can happen when we put too much trust in technology – and ignore the warning signs that something is wrong.
So what did go wrong with the Horizon system, and what lessons can we learn from it in our use of POS platforms today?
With technology, it’s people who matter most
The first thing to say is that POS technology has come a long way since the Post Office introduced Horizon. 25 years ago, software-based electronic POS networks were still in their infancy. It’s difficult to sum up in a few words just how much has changed in terms of how POS software (and software in general) is developed, tested and implemented, and how much better things are in terms of scalability, usability, reliability and security.
But safe to say, back in 1999, the Horizon system wasn’t up to the scale of the task asked of it. Yet it’s too easy to point the finger of blame at immature technology. The most serious errors were human.
Horizon was developed by the UK software company ICL, which was bought out by Japanese tech giant Fujitsu in 1998. Just a year later, winning the UK government contract to supply the Horizon system to the country’s 14,000 Post Offices was a major commercial coup for Fujitsu.
Creating a centralised POS network for so many branches would be a major undertaking even today. Back in 1999, it was the biggest non-military IT project in Europe. Yet it’s clear from today’s perspective that the resources allocated to it were totally inadequate.
The inquiry set up into the scandal heard that the development team working on the EPOS side of the system had just eight members. One member of the team alleged that up to half of his colleagues didn’t have adequate programming skills or experience for the scale of the task.
That’s a basic rule of every POS installation project, no matter how big or small. You have to have the right people with the right skills to configure and stage the system. Otherwise you are throwing the door wide open to problems further down the line.
Heed the warnings
The next major error came in testing. The inquiry again heard evidence that the Post Office was made aware of “severe difficulties” postmasters were having with the system in trials ahead of launch. But they ploughed on regardless. What’s the point of running trials if you don’t respond to issues that are thrown up?
Unfortunately for the sub-postmasters caught up the scandal, these mistakes were repeated over and over again in the decade and more that followed. The same member of the development team referenced above told the inquiry that, as early as 2001, he was aware of ‘hundreds’ of bugs in the system. As late as 2013, a well-respected professional consultancy called in to investigate the by-now overwhelming evidence that something was badly wrong with Horizon concluded that they could “find no evidence of system wide (systemic) problems.”
How could such a glaring miss occur? Well, one answer is that, in a way, the consultants were right – there may well not have been any obvious systemic problems. If the IT professionals and management above them were guilty of ignoring all the ‘hundreds of bugs’ in the early days, by 2013 those looking into the issue were guilty of assuming that ‘hundreds of bugs’ must add up to a major system-wide problem. In other words, they were looking for the wrong thing.
This reflects our tendency to see technology in very black and white terms – it either works perfectly, or not at all. There’s not much nuance in between. The UCL’s Information Security Research and Education department’s blog does a great job in illustrating this in its analysis of the technical issues that arose in the Horizon case.
In short, it’s easy to underestimate the technical complexity of digitising and automating transaction processing and logging in a distributed system, such as a centralised POS system that connects many branches. It only takes minor and momentary glitches – such as a loss of connection and synchronisation between parts of the system – for errors or abnormal data records to occur. In general, the probability of such errors occurring is low. But as the blog concludes, when you scale up the number of transactions – millions every day, in the case of the Horizon system – the number of actual errors inevitably goes up.
From that perspective, the problem with Horizon wasn’t that it couldn’t do the job it was designed for – 99.999% of the time it did. The real failing was a lack of appreciation that errors in such complex systems are inevitable, and having a robust set of protocols in place for dealing with them.
Today’s POS systems are incredibly sophisticated, agile and scalable. Not only do they operate at a rarified level of complexity in terms of the transactional feats they perform, they also increasingly form the hub of business-wide systems that automate management of finance, inventory, customer relationship management and more.
Still, if there’s one technological lesson to take from the Horizon scandal, it’s that we shouldn’t confuse even the most impressive performance levels with perfection. A key part of getting the best out of your POS or any other digital system is robust, fit-for-purpose performance monitoring, expecting and understanding anomalies when they occur, and having plans in place for resolving them appropriately.