We want to know how accurate a test sniffer dogs are. To put it another way: what are the rates of true positives and true negatives when we test people with sniffer dogs?
-
-
Show this thread
-
According to the police, the true positive rate for sniffer dogs is 80% - there is a 4 in 5 chance that an alert from a dog will result in the person having drugs or having recently been in contact with drugspic.twitter.com/4S004hPJwZ
Show this thread -
So what's the true negative rate i.e. the chance that someone who DOESN'T get stopped DOES have drugs?
Show this thread -
Well, this is extremely hard to know. Some research suggests it is close to 0% (!), because huge numbers of people get into music festivals with drugs But let's assume it's better, maybe at the same 80% figure we're using abovepic.twitter.com/BzKXWM1nBl
Show this thread -
So let's say that 100 people turn up to a music festival. Of these, 5 have drugs on them and 10 have recently been in contact with drugs
Show this thread -
For this type of test, we usually make a 4x4 grid to represent the rates of true and false positives, like so we can fill in the numbers at the end of the right straight uppic.twitter.com/RnDHZ9Ormf
Show this thread -
Now, our true positive rate - taken from the police - is 80%. That means that out of our 15 "has drugs" people, we'll identify 15*0.8 = 12 correctly, and 15*0.2 = 3 incorrectly, like sopic.twitter.com/e2SFzTJ6aY
Show this thread -
So far, all well and good. Lots of people correctly identified, not that many missed. Very few "false negatives" But wait, there's more
Show this thread -
Now, let's apply our "true negative" rate. We're assuming that the test is 80% accurate at identifying people who don't meet our criteria. So 85*0.8 = 68 correctly identified and 85*0.2 = 17 incorrectly identified This doesn't look so good!pic.twitter.com/s3dZwKWfbp
Show this thread -
Overall, the total number of people who test "positive" is 17+12 = 29, and the total "negative" is 3+68 = 71pic.twitter.com/LONF7eVjyI
Show this thread -
So using our (ENORMOUSLY GENEROUS - I'll get to this in a second) assumptions, the predictive value of the dogs is around 40%
Show this thread -
This gives you some idea of the issue - for every 100 people screened by sniffer dogs, a large volume of people will be found to have drugs who don'thttps://twitter.com/GidMK/status/1004527525551173632 …
Show this thread -
In fact, even if sniffer dogs are right 80% of the time, because taking drugs is less common than not taking drugs, you'll catch more INNOCENT people than guilty ones
Show this thread -
But...what if the dogs aren't right that much? What if, as many people argue, the dogs are wrong 50% of the time? Or 75%?
Show this thread -
Here's where it gets really problematic. Here's the table where the dogs are right 80% of the timepic.twitter.com/6IB2Jpy4KW
Show this thread -
And here are the tables for 50% and 25% - both of which have been observed in real lifepic.twitter.com/yOYKVtyggW
Show this thread -
As you can see, the predictive power goes down significantly. Even with 50% accuracy, only 1 in 5 of the people stopped will actually have drugs on themhttps://twitter.com/GidMK/status/1004530137872809984 …
Show this thread -
This is why when you are testing for an uncommon outcome, your test has to be REALLY GOOD Drug sniffing dogs, even by the best estimates, are not
Show this thread -
This goes doubly for screening large numbers of people. If you screen, say, everyone who goes into a music festival, you will likely catch few offenders but identify many totally innocent people
Show this thread -
So what should we do to prevent harm from drugs if we know sniffer dogs are ineffective? Tons of things. I'll make another thread about these options if anyone's interested
Show this thread -
Also P.S. I got a few pieces of terminology wrong - in particular, the first tweet should read "diagnostic test" not "intervention" - and also the "25% table" is slightly misleading as I've assumed that the negative predictive power is still 75% rather than 25%
Show this thread -
P.P.S TL:DR version is that drug sniffer dogs, even when they get it right 80% of the time, still make more mistakes than "hits" when not that many people have drugs
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.