Conversation

what do i doooo with my e.g. r=.09 correlations at 480k sample size? it's definitely a Real Correlation, especially when there's a cluster of similar correlations, and they're mildly interesting, but a little *too* mild if u know what im sayin

nimbly leaps 'round guards

@NLRG_it

Jan 24

Replying to

@Aella_Girl

how are you determining whether the correlation is real? are you compensating for multiple testing?

Aella

@Aella_Girl

Replying to

@NLRG_it

does that mean p-hacking? im not compensating because my sample is 480,000

3:52 AM · Jan 24, 2023

2,315

Views

Likes

nimbly leaps 'round guards

@NLRG_it

Jan 24

Replying to

@Aella_Girl

p hacking is doing it maliciously but its the same idea yeah. even with very large n, depending on how many correlations you look at, multiple testing can be issue. my recollections is that you have a lot of variables; abt how many correlations would you say youve looked at?

Replying to

and

If p=.05 is the significance threshold for one hypothesis tested, it's p=.025 for two, etc. So it depends on how many comparisons you try to make with your data.

Replying to

and

Having a big sample can reduce your p-values for small effects (like .09 correlations), but you still need to correct for multiple hypothesis testing. Unless you have lots of (real) significant results, Bonferroni correction is fine and easy (just multiply p by n).

Replying to

and

Multiple test bias isn't necessarily fixed by sample size. It's fixed by having a very low p value. The more p values calculated the more likely at least one will be below your cutoff by chance. Simplest correction is to multiply all p values by the number of tests.