Follow

Excuse this possibly silly question but:

Why is optional stopping in sample collection a problem, given the optional stopping theorem? (Or is it still considered a problem?)

https://pubmed.ncbi.nlm.nih.gov/24659049/

https://files.osf.io/v1/resources/m6dhw/providers/osfstorage/5d40ac7e26ebf5001984ae8d?action=download=1

@devezer Thanks! I guess my question was mostly: why is it a problem for frequentism at all? I know there are simulation results showing that optional stopping leads to inflated type 1 error rate (under NHST), but I was confused as to how the optional stopping theorem doesn't mean that this result is impossible.

After much, much discussion yesterday, I think the answer is that an NHST analysis on IID samples isn't a martingale. But I haven't quite figured out how. (My knowledge on martingales is fuzzy)

@devezer Right.

My problem is that the optional stopping theorem suggests that you can't win a fair bet with betting strategies that take into consideration all past information (but none future). E.g., the doubling strategy where you double your bet whenever you lose won't work.

I'm having a hard time seeing how optional stopping in study design isn't in a similar situation. Assuming IID, the researcher can't cheat the system without access to future information.

(fwiw I think my confusion is that IID doesn't guarantee the bet is fair, though I still don't yet quite understand how)

@UlrikeHahn @devezer No. I’m assuming the samples are drawn from a null distribution, which at 0.05 alpha should mean 5% false positives. This is how the one simulation study I found coded it.

“Winning” is entirely arbitrary. The phrase “can’t win in the long term” just means won’t deviate from the expectation (because the fair price of the gamble is defined by the expectation). So, in this case, we can define “winning” as committing exactly 5% false positives.

@kinozhao @devezer

I wonder if we are speaking at cross purposes: in my coin analogy, the probability at each toss remains .5 (and in that sense the distribution remains constant), but your actual probability of obtaining heads is different.

I'm trying to say that the effective distribution for whatever NHST statistic you are looking at will no longer match the assumed one...

Thanks! After much, much discussion yesterday, I think my confusion is that I assumed an NHST analysis on IID samples is a martingale (because it's IID, and I thought it would be analogous to CLT-style Monte Carlo resampling). My knowledge on martingales is pretty fuzzy because the concept of expectation is extremely difficult for me. Thanks for the references!

Berna Devezer@devezer@mastodon.social@kinozhao Shouldn't be a problem in Bayesian stats. I guess problems may arise due to misapplications? Like when it's indexed on statistical significance in a frequentist setup, for example. https://statmodeling.stat.columbia.edu/2018/05/02/continuously-increased-number-animals-statistical-significance-reached-support-conclusions-think-not-bad-actually/