Sunday, February 05, 2006

Random musings on experimental design

These thoughts are inspired by going over this article from Peter W Kalivas' lab (Neuropsychopharmacology, sub. appears to not be required) in preparation for my lab meeting tomorrow. However, this should not be considered a critique of the article or the persons involved--it may not, in fact, apply to this paper at all--they just happened to be the people who inspired these ruminations. This post gets prefaced by pointing out that I have not studied experimental design or the philosophy underlying said in any big fashion, and that my approach to both is probably fairly simple at best. Nevertheless...

As I understand it, the predominant paradigm in modern science is Popper's idea of falsifiability. There are two key aspects of this idea for this discussion (very simply put):

1. A scientific theory is defined as being one which is potentially falsifiable. That is to say, a theory is scientific if it can potentially be proven wrong in a reasonable fashion. Therefore, a model can be accepted by the scientific community to the extent of A: its explanatory power and B: its potential for falsification.

2. The ideal method--under this paradigm--with which to test a hypothesis is to prepare an experiment designed to prove a hypopthesis false. Thus, if you fail to falsify your hypothesis when testing it at its theoretical breaking point, you have more rigorously demonstrated that it has a high probability of truth (note that this does not make the hypothesis true: just that it is more likely to be true than if it was successfully falsified, or even if it had been tested under less rigorous circumstances).

Now, I cannot speak for non-neuroscience disciplines, because my knowledge of how they are studied in the modern era is relatively limited. But it seems from my experience of studying neuroscience and reading neuroscience papers that this approach, although not abandoned, is weak to a certain extent. Instead, the common phrase one hears is "one should put one's best foot forward." And although there is a strong fixation on making sure one does not try to muddy the waters by putting forward data which might be bad, there is also a strong emphasis on making sure one gets usable, publishable results.

I can think of a few reasons underlying this. For instance, one problem is that we're dealing with one of the most complex aspects of biological systems, which are inherently more complex than nonbiological systems (or so I would argue). Thus, an experimental design intended to falsify a hypothesis could easily succeed in falsification for the wrong reasons--for example, an attempt to make an experiment more rigorous by examining a structure a further step removed from the activity you're studying could result in falsification because a slightly alternate pathway is involved, or because the understanding of the connections between these two structures is just slightly flawed, or because a drug one is experimenting with is processed differently in one region than another. And, you begin to see, you quickly end up with results that falsify a hypothesis: but which hypothesis?

Another problem is the publish-or-perish model currently dominating the field, both in terms of funding, and in terms of careers. A successful falsification of a hypothesis that doesn't dominate a field is inherently less sexy than putting forward evidence supporting such a hypothesis. Anyone, after all, could come up with something wrong and demonstrate that it's wrong. The trick is coming up with something that's right. Similarly, the model leads to a rush to publication, which means you want to have the best data demonstrating the viability of a hypothesis as quickly as possible.

There's a lot more involved than this. And one should not take this as a claim that neuroscience, or biological science, or even science is inherently flawed and can't potentially be used as a tool for discovering knowledge about the universe. I wouldn't be here right now, and I wouldn't be writing this right now, if it was the case. It's as much an interesting piece of gristle that I'm chewing on right now as anything else.

Thoughts?

3 Comments:

At 11 February, 2006 19:12, Anonymous Anonymous said...

I'm pretty surprised you've made it most of the way through a Bachelor's degree in any science and not had a good run down on this topic.

Anyway, the hypothesis that most experiments try to refute is that there is no difference between two or more treatment groups. This is called the null hypothesis. If you can't refute the null hypothesis, you are unlikely to publish your results because, well, it could just be that you didn't measure enough samples or something. If you do refute the null hypothesis, you are at least able to say "these two (or more) groups were different, and it was probably due to this particular factor that we controlled for by the design of the experiment." And you can probably publish what you found.

But! It can be tricky, because maybe what you found was already known, and that would be boring (and less publishable). My understanding is that the least risky (but sometimes harder to design) experiments are those which will give some type of information whether or not the null hypothesis is rejected. This might be done by testing multiple parameters, or by testing parameters that can give unexpected responses. It can also make a seemingly unpublishable paper publishable, in the sense of, well, you can always start a controversy over the current paradigm if your well-designed experiment can't reject the null hypothesis that it supposedly should. This is actually pretty common in, say, medical journals, and is why most of the non-science-literate public gets so frustrated with scientists for "not knowing what foods are bad for you."

If you want, sometime I could explain in some detail to you how my graduate work satisfied the win-win style of experimental design (thought up by my ever-wise advisor, of course).

 
At 30 October, 2015 00:49, Anonymous Anonymous said...

reebok shoes, soccer shoes, ferragamo shoes, oakley, ray ban, louboutin, beats by dre, valentino shoes, iphone 6s plus cases, mont blanc, herve leger, bottega veneta, asics running shoes, nike air max, north face outlet, ghd, hollister, ralph lauren, chi flat iron, s5 cases, nfl jerseys, new balance, lululemon, vans, nike roshe, iphone 5s cases, iphone cases, baseball bats, ipad cases, vans shoes, mcm handbags, iphone 6 plus cases, instyler, p90x workout, gucci, babyliss, soccer jerseys, timberland boots, birkin bag, celine handbags, insanity workout, iphone 6 cases, abercrombie and fitch, jimmy choo shoes, iphone 6s cases, wedding dresses, converse, mac cosmetics, hollister, north face outlet, nike air max, giuseppe zanotti, coach outlet

 
At 11 December, 2015 02:11, Blogger mmjiaxin said...

nike free 5
michael kors canada
fitflops outlet sale
nba jerseys
coach outlet store
kobe bryants shoes
michael kors outlet online
christian louboutin outlet
ferragamo shoes
adidas wings shoes
coach outlet store
coach outlet
herve leger dresses
ralph lauren outlet
michael kors handbags
ralph lauren uk
giuseppe zanotti outlet
true religion canada
discount oakley sunglasses
michael kors handbags sale
mm1211

 

Post a Comment

Send Haloscan trackback ping

<< Home

Academics Blog Top Sites