John Protzko

How to completely ignore a research finding—rationally!!!

Posted on | November 5, 2015 | No Comments

Dear Researcher;

We understand that, occasionally, you come across some research finding that is…inconvenient. This evidence goes against your favorite hypothesis. It would be best if that evidence…went away.

The problem is, we are scientists. We can’t just pretend something doesn’t exist when faced with evidence.

Fear not, below you will find a menu of ways of dealing with evidence that goes against your favorite hypothesis. Each of these strategies is accompanied by a complete ratinoalization which you can use.

While there are more extreme approaches, those can sometimes be construed as ‘irrational’ or ‘unscientific’. Others cannot level such claims against the strategies below.

In addition to the justification for ignoring the research, the argument is also written in psuedo-Bayesian form, which allows you to use math and equations to show how extra-scientific the strategy for ignoring research is.

Strategy Example Description Bayes
Pretend it didn’t happen “I haven’t read that one yet” This strategy is a classic. Simply pretend that you haven’t read that bit of evidence. When asked about it, you can pretend you haven’t come across it. This strategy must be used sparingly. If used too often, people start thinking you may not be serious about keeping up with the literature

 

 p(ht1) = p(ht2)
Doubt magnitude of effect size “Right, but the effects are so small” This strategy can be used in response to meta-analyses which often produce small, yet statistically significant findings. The strategy is to ignore the effects a finding may have on the logic of your position and to instead attack the magnitude of effect size. This can only be used in small effect size findings.

 

(p(d│h))/(p(d))≈1
So that
p(h)≈p(h|d)
Create unknown null studies “Right, but I wonder how many unpublished null effects there are…” Once a new study comes out which goes against your favorite hypothesis, immediately create a number of new, ‘unknown’, null findings which ‘must’ also be out there. These size of these studies must walk a fine line between being adequately powered to as not count as severely underpowered, yet not powered enough to plausibly contain a significant effect. This strategy can be dangerous. Make sure not to create too many fictitious null findings. Otherwise, a new study could end up lowering your belief!

 

d1╞ d2, d3…di such that d1 = -∑ d, so that p(h) = p(h|d1…di)
Claim it is not evidence “No, no, no, that study isn’t evidence!” This can often be done by denying a fundamental class of evidence (e.g. Bayesian, falsificationist, incremental).

 

p(d|h) = p(d)
Split “Oh yes, I have heard of that…” This strategy is a long-time favorite. Once learning a new, inconvenient finding, split your current beliefs into two, one which contains the new evidence and one which doesn’t. You then continue learning new information, altering both beliefs. This is done until the first bit of evidence comes along that is inconsistent with your “not-really-updated version” of beliefs. Claim that new evidence ‘completely falsifies’ the entire line of research, and resort back to your original, untainted beliefs.

 

p(h)
p(ho) p(hU*)
Question replicability “That finding doesn’t replicate” This is a great technique. Simply claim that the to-be-ignored finding does not replicate. Be warned. As journals become more willing to publish well-powered failures to replicate, this will be harder to pull off without citations. Even now researchers are becoming savvy and asking for who tried and failed to replicate a finding.

 

(p(d│h))/(p(d))=1 So that p(h)=p(h|d)
Actually reverse evidence “Actually, that study is really in favor of the hypothesis it tries to refute!” This strategy is the most complicated, requiring the linguistic skills of the best philosophers and theoreticians. With great skill, an inconvenient finding can actually be turned into evidence for you preferred hypothesis, not against it! Do not attempt this without serious practice and training.

 

 (p(d│h))/(p(d))=1/((p(d│h))/(p(d)))

Possible citation includes Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: the effects of prior theories on subsequently considered evidence.Journal of personality and social psychology, 37(11), 2098.

Warmly, and with tongue in cheek,

Protzko

Comments

Leave a Reply





Search