Algorithms are good

I have read a few articles recently about algorithms. I think someone has written a book about them, and usually the articles have been scare stories, talking about how dangerous algorithms are and how they are taking over our world. Well, I disagree. I love them.

What is an algorithm? The free online dictionary defines it as a step-by-step problem-solving procedure, especially an established, recursive computational procedure for solving a problem in a finite number of steps. I think we had algorithms before computers, but the word has gained currency in the last half century as computers have become more prevalent.

Essentially, an algorithm is a way of modeling complexity. Any problem that has a lot of available data, and where a model or set of rules can help to make sense of the data can benefit from an algorithm. As computers have gained power, so algorithms have been able to become more complex and process more data.

I see a clear parallel between a computer algorithm and the models, assumptions and stereotypes we all use to simplify our lives. Should I cross the road here, or there? Our brain goes through lots of steps very quickly. How tired am I? How far would the diversion be? Can I trust the signal “walk” signal? How courteous are drivers around here? Am I wearing visible clothes? Do I have kids with me? How much of a hurry am I in? Are there any shady looking characters about? The brain will process all these questions and make a decision for us. In “Rain Man”, the autistic Dustin Hoffman found this sort of complex processing of uncertainty too difficult, and made a mess of the challenge. But most of us can judge the risks and subtle choices quite well. The human brain is immensely powerful.

What has happened is that computers are gradually catching up with our brains, through sheer force of data. This is something to celebrate, not fear.

When I look back at my career, algorithms have played a major part. Working at the boundary of strategy, marketing and analytics, I have faced many challenges where a slightly better model of complexity could lead to competitive advantage and business value. I was lucky to do this at a time when we were facing many of these challenges from first principles, and I found them very interesting.

My first paid job was for a company trying to develop software to help logistics companies plan their depots and truck routes. It was called Vanplan, but our software did not really deal well with the vast data and complex variables involved so it did not sell well, and wags in the company rechristened it Truckfuck. Inside Vanplan was a database of all the main junctions and roads in the UK and their average speeds. We had to build this from scratch using maps, and paid part-timers to create this. There was an elegant algorithm, that I tried to improve and run faster, since the computer power at the time was a major constraint. Then Vanplan had other related algorithms, which also needed tweaking to make them reliable and faster and to deal with real-world variables like rush hours and driver habits. It was fascinating.

It is hard to credit that this was only thirty years ago. Now Google have coded the same data globally with far more accuracy, and the equivalent of Vanplan is offered online as a cheap application, computing in a second what used to take us hours. I call that progress.

I can quote many other examples. I developed or interpreted or simply applied many algorithms. I allocated stations to sales reps, defined sales rep territories, optimized depot structures, and sought locations for new stations. I tried to model petrol station pricing, and optimize the location inside stores for categories and items. I built models of costs and incomes, for my own firm and competitors. I tried to model an insurance scheme whereby we would guarantee an engine cold-start performance, based on weather patterns and where trucks typically were parked overnight. I tried to model the financial impact of alternative advertising media and campaigns.

All of these required algorithms. They were imperfect, having to model data and assumptions, and were limited by the computing power available at the time. It was fun. I was blessed, in the right place at the right time with the right skills and experience and motivation.

The only thing that has happened since is that data has become more widely available and computers have become immensely more powerful. This means the algorithms are better, more reliable, and more usable in more areas. Good.

Actuaries always tried to assess pension risks and returns. Now they can do it more accurately. Banks always tried to assess loan risks. Years ago it was based on the biases of the bank manager, now they have better systems and data. The police always used models to try to catch criminals, choosing Friday nights to hang about outside pubs. Now they have wonderful profiling and contextual algorithms. Intelligence agencies always tried to find terrorists and spies, with comical ineptitude. Now they are rather better, even if still as dishonest. Brokers always tried to pick stocks, based on dubious knowledge and bias. Now they have data to spare. Doctors always guessed ailments based on symptoms and test results. They still, do, but have become much better at it. When I was interviewed for Shell in 1982, I got in based on a combination of science, prejudice and luck. The same three factors would apply today, but the science would weigh far more heavily. Finally, traffic planners always tried to position lights and lanes to keep traffic moving. But recently, on one miraculous morning, I drove from Forest Hills to the Queensborough bridge through a total of 62 lights, and stop at just one of them.

This all can make our lives better. We have GPS, we have wonderful knowledge sources, we can shop around to get good coverage and prices based on our own habits and characteristics. If we have nothing to hide, we can feel safer with the smarter policing. We can expect more medical breakthroughs. We can still use our brains to make even better personal models and use our own computing power to test them. Algorithms are good.

So why all the fuss?

Partly we are scared of things we don’t understand. Partly it is journalists and authors looking for a story. Partly we are trained by sci-fi to be scared of computers and big brother. And there are one or two consequences of algorithms that do require control.

First, the finance world needs some brake on velocity of trading. If everyone has similar algorithms, you risk events like the flash crash of 2010. My solution is simple: charge a Tobin tax on transactions. Built into algorithms, that on its own would reduce velocity of trading, as well as creating many other benefits. Problem solved.

Controlling big brother is nothing to do with algorithms, and everything to do with law and transparency. If the spooks had to own up to what they were doing, they might still do it, but they would need stronger justifications, and we could exercise some choices too, such as not using US technology companies.

The final issue that I can see is excess discrimination by segment. It is good that groups that offer better risks should have cheaper car or medical insurance, but somehow everyone must have affordable insurance. It is good that the police can target neighbourhoods and certain demographics, but it should not be cost-free, and people need an escape route from a disadvantaged segment.

Most of this is about smart welfare. Obamacare helps, but is not enough. Imagine someone with a criminal record and former addiction: their route to health and honesty is hard enough already, without imposing all the added costs that algorithms would calculate. So let the providers use the algorithms and charge the higher costs (though don’t let them refuse coverage) and then make sure the overall welfare package offers a fair chance and great incentives and enough support, through clinics for example. Unfortunately, this line of thinking is hardly mainstream in the USA, even though it would certainly pay for itself.

What about black youths whose lives are made a misery from police harassment? Well how about the police having to pay a token amount per search when no suspicious behavior is discovered? The payments could be dependent on other socially desirable actions, such as looking for work, no truancy by kids, and up-to-date maintenance payments by single fathers. That way everyone gets an incentive to behave better, police included, while the power of the algorithm can still be utilized. Hopefully, succeeding generations of algorithms will be so smart that more and more innocent sub-groups can be eliminated from target populations.

So, let us not be scared of algorithms, let us embrace them. The world would be a less efficient place without them. But, in a few specific cases, let us think about the possible consequences, and put humane remedies in place.