Conversation

Representative samples may be overrated. A replication of 27 survey experiments (n = 101,745) finds that convenience samples overwhelmingly produce similar findings to representative samples. pnas.org/doi/full/10.10 In 0 of 393 analyses were the effects in opposite directions.

561

2,217

Aella

@Aella_Girl

Replying to

@jayvanbavel

ty for everyone tagging me this is confirming my biases and i hope it's also accurate

5:52 PM · Jan 5, 2023

4,627

Views

Likes

Random Academic

Replying to

and

Might be interesting to: Find a statistic that has been well-studied using (as much as possible) representative samples, such as number of partners, practices, etc. Then do an online poll. See how much agreement you get.

Replying to

and

I do this constantly in my surveys to check my deviation from general pop

Replying to

and

It applies only if the treatment effect heterogeneity is low, which is the case in the political studies tested but might not generalize to other areas of life.

Replying to

and

Unfortunately I think the paper is only looking at causal effects (so you're probably fine running experiments on convenience samples, but that doesn't mean convenience samples will show you the same true population values for a statistic that a representative sample would).