Conversation

Karina Nguyen
@karinanguyen_
Harmful language models like dangerous viruses spread easily, but are very challenging to control once out of the lab. We must hold their distribution/deployment to the same rigorous ethical standards — for that we need tighter collab b/w AI safety researchers + policymakers.
Quote Tweet
Lauren Oakden-Rayner
@DrLaurenOR
·
This week an #AI model was released on @huggingface that produces harmful + discriminatory text and has already posted over 30k vile comments online (says it's author). This experiment would never pass a human research #ethics board. Here are my recommendations. 1/7 twitter.com/ykilcher/statu…
Show this thread
I agree with KCramer. There is nothing wrong with making a 4chan-based model and testing how it behaves. The main concern I have is that this model is freely accessible for use. While open science is a great principle, I'm a medical doctor and safety researcher by training and we always need to consider possible harms. Human research ethics is baked into the very foundation of our field, because of a long history of human rights abuses in the name of science, in particular experiments that cause harm to disempowered and marginalised people without their consent. It should be clear that this model carries a significant risk for this sort of harm, given the fact such an experiment has already been performed. The model author has used this model to produce a bot that made tens of thousands of harmful and discriminatory online comments on a publicly accessible forum, a forum that tends to be heavily populated by teenagers no less. There is no question that such human experimentation woul
Text from huggingface discussion: Given the demonstrated risk of harm, this model should not be freely accessible. The medical community has well established guidelines on how to manage the sharing of research materials which involve a risk to human subjects, with data privacy being the most common risk. It is common to allow research access to datasets in this context via a registration platform, where the applicants who are seeking access must describe their proposed research, and sign an agreement for data use. See the NIH/TCIA and MIMIC datasets for examples. The latter even has a requirement for applicants to pass a course in human research ethics prior to obtaining access to the data. A similar system should be in place here, and be used as the template for future model sharing where the model has the potential to produce harm.
3:54 AM · Jun 7, 2022Twitter for iPhone
4
Retweets
16
Likes