Conversation

critiques i get from ppl who don't know stats: "Selection bias! Internet surveys aren't 'studies'! Just Aella's Audience!" from ppl who do know stats: "You should try [different new statistical method] instead, would account for [obscure math thing I need to go learn now]"

458

Replying to

The results are interesting and fun. But most readers make inferences about the broader population which makes selection bias is an issue. Saying it's only "Aella's audience" helps but doesn't solve it: it's some subset who choose to answer for diverse and unknown reasons.

Aella

@Aella_Girl

Replying to

@erikbryn

this would piss me off less if I saw the same critiques directed as much towards the majority of published, peer reviewed academic research that comes from surveys, which have *way* less representative samples than I do. The discrepancy in critique makes me think I'm targeted.

4:34 PM · Jan 31, 2023

6,166

Views

Replying to

and

And also, to clarify, the biggest survey I'm working on (500k people) is only around 5% my audience; the rest is from people who took the survey through tiktok and probably have never heard my name before.

Show replies

Keiran of woolly mammoths

Replying to

and

How much do you know about your audience? As I’m sure you’re aware academic studies often go out of their way to characterise study populations, so they can do fancy stats things. Do you have more info than “follows aella”?

Replying to

and

Yes! I both have been studying my own audience for many years (to the degree I am very good at predicting how to split questions in twitter polls for closer to 50-50 results), and I also include really comprehensive demographic questions in many of my surveys.

Show replies

Tailcalled (tailcalled@schelling.pt)

Replying to

and

Have you tried describing some peer-reviewed research to your followers to double-check that they have different reactions to that?

Replying to

and

maybe I should pretend I did some research that's actually directly lifted from something published to test this

Show replies

Replying to

If it makes you feel better, I get this critique from reviewers in almost every paper I write using survey data. And I use a similar response: it is what it is -- still interesting and even useful even if it may not be entirely representative.

Replying to

and

One concrete thing you can do that helps (but won't fully satisfy critics) is to compare respondents to non-respondents on observables like gender, age, etc. that you may be able to obtain from other sources. The other thing is not to sweat it too much - surveys are fun.

Show replies

Replying to

and

You have an incomprehensibly large number of anti fans for not being a top twitch streamer or YouTuber

Replying to

and

"isolated demand for rigor" is a nasty trick

Replying to

and

my guess is: most people are unaware how much we depend on surveys, and how much goes into designing one well / interpreting results meaningfully so maybe not targeted, you just have the misfortune of being their first time grappling with the idea of surveys as research