Conversation

One thing that bothers me is when ppl critique my data as "unscientific" due to sampling bias. First, y'all better not look into a lot of what you think are "scientific" studies Second, there's nothing wrong with sampling bias if you acknowledge limitations and don't extrapolate.

276

Aella

@Aella_Girl

Jan 3, 2021

Yes, I know most of my responders are white cis men between 25-40 who are associated with STEM, vaguely liberal/centrist/libertarian. There's nothing about collecting data from a particular demographic that's unscientific as long as you're upfront about the demographic.

Replying to

The problem isn't that they are "white cis men etc.". The problem is that you only have answers from people who are motivated enough to give you answers. And you have absolutely no idea how different it is from the general population unless you have an ability to sample the

Replying to

and

general population without this bias. I.e. if you ask people opinion on X and get results, you only get results from people who are actively thinking about X. You don't get responses from people who are just not interested, or maybe never heard, etc. So the problem is that

Replying to

and

the sample is influenced by the question itself. There's a simple way to address that - ask people to commit to reply before they hear the question. E.g. "I'll give you $10 to answer my question". And only after they agree you show them the question.

Replying to

This would incentivize ppl to say yes for the $10 without being accurate on the test, would select for ppl who need $10 more. You wouldn't get any busy right people. There's no way to remove "motivation to answer the question" from answers to the questions.

Replying to

and

*rich people

Replying to

The difference is that the bias of paid surveys is well understood - it excludes rich people, yes. Also many studies are done on college students, and it is a somewhat well-understood and meaningful demographic. If you use internet volunteers, however, you don't understand the

Replying to

and

sample bias you have. It can be anything. It can be arbitrary bad. For example, suppose you ask about attitude towards kinks on a kink forum. The answers might give you no information whatsoever on the attitude toward kinks within the general population.

Aella

@Aella_Girl

Replying to

@killerstorm

I mean in that example it would be pretty easy to tell the limitations. Obviously kink forums aren't representative of kinks

7:57 PM · Jan 3, 2021

Replying to

My point is that you that limitations of twitter polls are not well-understood. It might be that your followers have very particular set of biases different from the same demographic in the wild. No way to know that without actually comparing.