Random musings on experimental design

These thoughts are inspired by going over this article from Peter W Kalivas' lab (Neuropsychopharmacology, sub. appears to not be required) in preparation for my lab meeting tomorrow. However, this should not be considered a critique of the article or the persons involved--it may not, in fact, apply to this paper at all--they just happened to be the people who inspired these ruminations. This post gets prefaced by pointing out that I have not studied experimental design or the philosophy underlying said in any big fashion, and that my approach to both is probably fairly simple at best. Nevertheless...

As I understand it, the predominant paradigm in modern science is Popper's idea of falsifiability. There are two key aspects of this idea for this discussion (very simply put):

1. A scientific theory is defined as being one which is potentially falsifiable. That is to say, a theory is scientific if it can potentially be proven wrong in a reasonable fashion. Therefore, a model can be accepted by the scientific community to the extent of A: its explanatory power and B: its potential for falsification.

2. The ideal method--under this paradigm--with which to test a hypothesis is to prepare an experiment designed to prove a hypopthesis false. Thus, if you fail to falsify your hypothesis when testing it at its theoretical breaking point, you have more rigorously demonstrated that it has a high probability of truth (note that this does not make the hypothesis true: just that it is more likely to be true than if it was successfully falsified, or even if it had been tested under less rigorous circumstances).

Now, I cannot speak for non-neuroscience disciplines, because my knowledge of how they are studied in the modern era is relatively limited. But it seems from my experience of studying neuroscience and reading neuroscience papers that this approach, although not abandoned, is weak to a certain extent. Instead, the common phrase one hears is "one should put one's best foot forward." And although there is a strong fixation on making sure one does not try to muddy the waters by putting forward data which might be bad, there is also a strong emphasis on making sure one gets usable, publishable results.

I can think of a few reasons underlying this. For instance, one problem is that we're dealing with one of the most complex aspects of biological systems, which are inherently more complex than nonbiological systems (or so I would argue). Thus, an experimental design intended to falsify a hypothesis could easily succeed in falsification for the wrong reasons--for example, an attempt to make an experiment more rigorous by examining a structure a further step removed from the activity you're studying could result in falsification because a slightly alternate pathway is involved, or because the understanding of the connections between these two structures is just slightly flawed, or because a drug one is experimenting with is processed differently in one region than another. And, you begin to see, you quickly end up with results that falsify a hypothesis: but which hypothesis?

Another problem is the publish-or-perish model currently dominating the field, both in terms of funding, and in terms of careers. A successful falsification of a hypothesis that doesn't dominate a field is inherently less sexy than putting forward evidence supporting such a hypothesis. Anyone, after all, could come up with something wrong and demonstrate that it's wrong. The trick is coming up with something that's right. Similarly, the model leads to a rush to publication, which means you want to have the best data demonstrating the viability of a hypothesis as quickly as possible.

There's a lot more involved than this. And one should not take this as a claim that neuroscience, or biological science, or even science is inherently flawed and can't potentially be used as a tool for discovering knowledge about the universe. I wouldn't be here right now, and I wouldn't be writing this right now, if it was the case. It's as much an interesting piece of gristle that I'm chewing on right now as anything else.

Thoughts?