Bots have been a huge threat data integrity in recent years, and I can't believe that bot protection is not yet a standard part of the data integrity section of IRB submissions. Gone are the days where "checking the quality of the data every few days" will suffice (2/n)
-
-
Prikaži ovu nit
-
Adding protections INTO your survey may take time and energy spent coding & creating advanced branch logic but it will save you hundreds of hours (and LOTS of money) if you do it right! (3/n)
Prikaži ovu nit -
A bit of my story: within 12 hrs of going live I had over 350 false respondents in my study. I tell you this now, but it took me hundreds of hours to identify these bots and more than a week's worth of work. I'm lucky enough to have a quant background which made this easier (4/n)
Prikaži ovu nit -
It took 10 codings schemes to reveal bots. If I hadn't had open ended questions, I am confident that I would not have identified the bots in my study. So, here are my lessons learned. Lesson 1: REQUIRE open-ended responses (5/n)
Prikaži ovu nit -
Lesson 2: Everyone doing online data collection needs to build in ***complex and advanced*** logic/inattentional checks throughout the first sets of surveys (and do NOT cluster them) (6/n)
Prikaži ovu nit -
Lesson 3: Add "honeypot" items to your survey. These are fields that are hidden to your average participants but are visible to bots. Name the items in an identical fashion to your other fields to prevent the bots from catching on (7/n)
Prikaži ovu nit -
Lesson 4: Captchas are not enough. But add them in anyway Lesson 5: Screen participants and then email those who passed the screener with a unique survey link. This takes more time, but you have to do it. NO PUBLIC LINKS. EVER. DON'T DO IT. (8/n)
Prikaži ovu nit -
Lesson 6: flag/prompt participants who are "speeding" through materials Lesson 7: Ask similar questions at different points in your study to check for inconsistencies (e.g., ask gender twice) (9/n)
Prikaži ovu nit -
Lessons 8-10: Your study will still have bots. Check your data. A LOT. Don't blame yourself. Acknowledge this as a historical factor influencing data integrity and prepare for it. That's it for now. Thank you for coming to my
#TedTalk. (10/10)#AcademicTwitter#DataSciencePrikaži ovu nit -
One last tip - don’t automate participant payment. Thankfully I did not. This allowed me to review my data before paying to avoid compensating bots. Definitely check all of your data integrity markers before compensation and have an IRB approved protocol to determine who to pay
Prikaži ovu nit -
Clarification: this study did not use MTurk
Prikaži ovu nit -
I'm adding a recent article I wrote describing how I handled and identified bots in my study for those who continue to find this thread:https://behavioralscientist.org/how-to-battle-the-bots-wrecking-your-online-study/ …
Prikaži ovu nit -
And one more. Just hoping to keep all of the related things in one place for those interested! :)https://www-new.statnews.com/2019/11/21/bots-started-sabotaging-my-online-research-i-fought-back/ …
Prikaži ovu nit
Kraj razgovora
Novi razgovor -
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.
| 