quarantine 'em

Humans build the training data sets. Humans pick which data sets to use in the training. Humans implement the training algorithms. Humans decide *when* the machine is adequately trained. Humans install the machine. Humans decide whether to honour the machine's decisions

So there you are, wrestling futilely with a system which persistently denies that you are who you are, while the engineers and managers who put you and this machine in the same room in the first place never enter your mind and in any case cannot be reached for comment

Are you with me so far? This is all humdrum so far. Bias creeps in, undesired behaviour oozes out. Humans make errors and the machine amplifies those errors Now the ugly part. What happens when we subtract good faith from this picture?

What if you WANT to create a biased machine? What if, based on everything I've just said, you've realised, correctly, that a biased machine is a highly effective, highly deniable way to deliver your bias?

In Mickens' "Tragedy of Darth Tay the Bigoted" the causes and effects are incredibly stark and easy to unthread. If an attacker has control over the A.I.'s training data, of *course* the resulting A.I. will misbehave

So what happens if any of the humans listed below is an attacker? And the system they're attacking is the very important real-world system which they are all nominally working together to automate? The criminal justice system or the healthcare system?https://twitter.com/qntm/status/1030837348630708224 …

Mickens gives a high-level goal of computer security as "ensuring that systems do the right thing, even in the presence of malicious inputs". In a machine learning scenario, inputs come from a whole lot of places which are not the user

Or, of course, you can secretly replace the "A.I."/"machine decision-maker" with a highly biased human or collection of humans Yeah, you were randomly selected by this button which I press when I want the machine to randomly select you