Odds are, it’s wrong

That’s the title of a very interesting article at Science News, arguing that bad statistics is the dirty secret of science. I believe it is.

Though there is an entire field of physics called “statistical mechanics”, the statistics there don’t go beyond the 19th century. To this day, physics undergraduate and graduate programmes cover statistics minimally, or not at all. Perhaps it seems unimportant to theorists, but it is crucially important in testing hypotheses, which is what experiments claim to do. Or perhaps hypotheses in physics are sufficiently clear-cut, and experimental data sufficiently clean, that sophisticated hypothesis testing is not necessary.

In other fields, hypotheses are murky and plentiful, data are noisy and ambiguous, but the practitioners are still ignorant of statistics. When the field is medicine, and the question is of new drugs or therapies, it is a crucial matter.

Famously, Sir Roy Meadow — creator of the discredited “Munchhausen syndrome by proxy” hypothesis — sent several mothers to jail with his expert evidence based on bogus statistics. The consequences of bad statistics may not always be equally bad, but if the medical literature is as riddled with them as recent articles suggest, the cumulative effect may be worse than anything Meadow did.

But bad statistics in the medical literature is just the starting point: there are problems throughout the practice of standard medicine. This is why, though I respect mainstream medicine and regard much “alternative medicine” as fraudulent and the rest as of very limited (and unvalidated) applicability, I was sufficiently annoyed by this post by Orac to leave this comment. (See also other comments there on dubious practices in the health industry.) I think Orac does, in theory, a great service by pointing out peddlers of pseudoscience and exposing their ignorance and, often, fraudulence. In practice, he preaches to the converted and, I suspect, antagonises nearly everyone else.

Advertisements
Next Post
Leave a comment

3 Comments

  1. You will be pleased to note that a basic course on Probability and Statistics is part of the Core that is done by all students at IISER.This at least introduces them to the possibility of methods affecting the outcome of a measurement (Bertrand's paradox). Whether they will recognise it when they see it during their research is another story; after all even statisticians and probability theorists can sometimes be fooled!

    Reply
  2. I remember reading up a bit on Munchausen by proxy a long while ago when there was a spam flood for "rescue Aditya Chandran" on the web and from that site and the best of my memory it sounded a bit iffy by itself; and especially doubtful applied in that case which sounded like a bad divorce going hellishly wrong for the lady concerned.thanks,Jai

    Reply
  3. Kapil — I am currently reading Jaynes' "Probability theory: the logic of science", where he basically trashes "orthodox statistics" (it's an unsystematic collection of ad-hockisms, a hodgepodge of the feuding schools of Fisher and Neyman-Pearson, according to him) and argues that physicists and astronomers have been better off not using statistical hypothesis tests because their own intuition is much better at assessing significance. He argues strongly for Bayesian methods as outlined by him; the methods go back decades and he gives huge credit to Harold Jeffreys and his 1939 book on probability and statistics. Today the worst of the polemics between the "frequentist" and "Bayesian" camps are over and Bayesian methods are widely accepted because they work, but I suspect the deeper points that Jaynes makes are still not widely accepted, perhaps because so many people would be personally hurt by it. For example, he says the chi-square test is often wrong by orders of magnitude because it makes incorrect assumptions about the tails of normally-distributed data. There are several other examples in each chapter.I am very far from finishing the book, much less absorbing it, but will blog about it one of these days.Jai — yes, that case seemed extremely odd, and like you, it seemed to me that the lady was being victimised.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s