quarantine 'em

This makes questions like "Why did the machine decide to grant this person a loan but not that one?" and "Why did it flag this person for investigating but not that one?" and, most importantly, "HOW DO WE STOP IT DOING THAT?" unanswerable

Thirdly, transference of responsibility. Do you see the mistake I made in the preceding few tweets? I talked about the machine's bad decisions, the machine's inscrutable behaviour, the machine's implacability. A machine is not responsible. A machine is installed by a human being

Humans build the training data sets. Humans pick which data sets to use in the training. Humans implement the training algorithms. Humans decide *when* the machine is adequately trained. Humans install the machine. Humans decide whether to honour the machine's decisions

So there you are, wrestling futilely with a system which persistently denies that you are who you are, while the engineers and managers who put you and this machine in the same room in the first place never enter your mind and in any case cannot be reached for comment

Are you with me so far? This is all humdrum so far. Bias creeps in, undesired behaviour oozes out. Humans make errors and the machine amplifies those errors Now the ugly part. What happens when we subtract good faith from this picture?

What if you WANT to create a biased machine? What if, based on everything I've just said, you've realised, correctly, that a biased machine is a highly effective, highly deniable way to deliver your bias?

In Mickens' "Tragedy of Darth Tay the Bigoted" the causes and effects are incredibly stark and easy to unthread. If an attacker has control over the A.I.'s training data, of *course* the resulting A.I. will misbehave

So what happens if any of the humans listed below is an attacker? And the system they're attacking is the very important real-world system which they are all nominally working together to automate? The criminal justice system or the healthcare system?https://twitter.com/qntm/status/1030837348630708224 …