Conversation

Aella
@Aella_Girl
Completely agreed with Spencer here. Probably the thing I've learned the most throughout my thousands of questions and feedback is exactly this - and despite my experience, I still find it very hard to guarantee question design that definitely won't hit any of these problems. 1/
Quote Tweet
Spencer Greenberg 🔍
@SpencrGreenberg
The exact wording in polls, surveys and psychology studies matters more than most people seem to realize. This is a big deal if you want to learn from polls and academic papers, or if you conduct studies yourself. Here are some dramatic real examples:
Show this thread
8
170
Aella
@Aella_Girl
I've also had the experience of reading about studies or measures, and then when I see their actual questions I am shocked at how easily misinterpretable they are. Regularly they use questions I would never ever use in my own research. 2/
2
51
Aella
@Aella_Girl
I thought for a while that they must have their reasons, it's gotta make sense somehow, they can't be just that bad at question design. But lately I'm starting to suspect they just outright *are* bad at it? Which freaks me out a bit and feels like it has huge implications. 3/
3
56
James McDaniel
@JamesMcDaniel
Replying to
@Aella_Girl
I wish initial research results would somehow stay “this is just one data point, gotta wait for others to independently replicate it” humble, instead of trumpeted to the general public right out of the gate as so often happens
4
Modern Lars
@larspetrus
Replying to
@Aella_Girl
Hot take 1: If answers depend so much on phrasing, maybe the thing you try to measure isn't that well defined. Hot take 2: Someone should do the research *on* how question phrasing affects answers. Might be a fresh new research field to found!
1
3
Snowie 1969
@1969Snowie
Replying to
@Aella_Girl
I reckon that when work is published and peer-reviewed, people will bring up the wording of the questions and criticize the conclusions the researchers have taken from those answers.
1
Birds 🐦
@Birdsbirdbird
Replying to
@Aella_Girl
People do talk about this, all the time... this is a huge, well known issue in survey design. Your lack of exposure to that isn't reflective of either the academic landscape or the polling industry (not to suggest I think either does a good job, but they do talk about this).
4