Conversation

Aella
@Aella_Girl
critiques i get from ppl who don't know stats: "Selection bias! Internet surveys aren't 'studies'! Just Aella's Audience!" from ppl who do know stats: "You should try [different new statistical method] instead, would account for [obscure math thing I need to go learn now]"
35
458
Erik Brynjolfsson
@erikbryn
Replying to
@Aella_Girl
The results are interesting and fun. But most readers make inferences about the broader population which makes selection bias is an issue. Saying it's only "Aella's audience" helps but doesn't solve it: it's some subset who choose to answer for diverse and unknown reasons.
4
28
Aella
@Aella_Girl
Replying to
@erikbryn
this would piss me off less if I saw the same critiques directed as much towards the majority of published, peer reviewed academic research that comes from surveys, which have *way* less representative samples than I do. The discrepancy in critique makes me think I'm targeted.
21
76
Erik Brynjolfsson
@erikbryn
Replying to
@Aella_Girl
If it makes you feel better, I get this critique from reviewers in almost every paper I write using survey data. And I use a similar response: it is what it is -- still interesting and even useful even if it may not be entirely representative.
1
5
Erik Brynjolfsson
@erikbryn
Replying to
@erikbryn
and
@Aella_Girl
One concrete thing you can do that helps (but won't fully satisfy critics) is to compare respondents to non-respondents on observables like gender, age, etc. that you may be able to obtain from other sources. The other thing is not to sweat it too much - surveys are fun.
1
5
Aella
@Aella_Girl
Replying to
@Aella_Girl
and
@erikbryn
tbf i might rage even more if the dumb responses were coming from reviewers?? like shouldn't they be aware that surveys ofc have limitations and as long as you're not making claims beyond the data its' fine and good? tho at least it's private.
2
2
Show replies