Skeptical science and medical reporting (#Scio13 wrap-up)

Ivan Oransky and I moderated a session last week at ScienceOnline, the yearly conference covering all things at the intersection of science and the internets. We discussed the topic ““How to make sure you’re being appropriately skeptical when covering scientific and medical studies.”

We started out discussing some of the resources we’d put up at the Wiki link. Ivan teaches medical journalism at NYU, and noted that he recommends these criteria when evaluating medical studies. I noted I use similar guidelines, and as a scientist, think about papers in a journal club format before I cover them on the blog, considering their strengths and weaknesses (especially in study design and analysis). Ivan also mentioned the need sometimes to consult a real statistician if you don’t understand some of the analyses–suggesting to “keep a biostatistician in your backpocket” or, failing that, to reach out to those at you local university, as “they tend to be lonely people anyway.” (Just kidding, biostats friends and colleagues!) A number of stats references for journalists were also mentioned (see the Storify for specific links). From here, we handed the discussion over to the audience.

One of the first topics we reviewed was just what is meant by being “appropriately skeptical,” which was a theme of the session that we kept coming back to. How does one do that without being an asshole? The importance of criticizing the study’s limitations and weaknesses–and not necessarily being a jerk to the authors–was noted. No study is going to be perfect, after all. It was also pointed out that anyone reporting on the study should go beyond the press release, and not to do so is in fact “journalistic malpractice.” Bora also started an interesting tangent–are medical studies more likely to be fake (or more deserving of skepticism about results) than more basic science reports? Also, is it worth reporting on bad studies? Sometimes this can help to point out the bad science (like that recent mouse-GMO study, which was reported on–negatively–in many venues). This recent study on “out” versus “closeted” homosexuals in Montreal was also brought up by Annalee Newitz–a small study that was widely reported, but was it designed and powered correctly to examine the questions it supposedly answered? (I haven’t read it, but just looking over the article, looks like “no”.)

Audience members also asked how to find sources to comment on studies. Ivan has previously written a post on this, and others in the audience recommended looking at other references in the story itself, or looking at reviews or meta-analyses on the topic to see who else may have expertise in these particular areas. However, SciCurious also noted that you need to be somewhat skeptical of those as well, and examine if the authors of these reviews or analyses have their own biases that may skew the information being presented.

The idea of “Glamour Mags” was also introduced. How should those reporting on a story know whether the results were published in a “good” journal or not? Several pointed out that just because a study is in a lower impact-factor journal doesn’t necessarily mean it’s not to be trusted. Eli elaborated, noting that fraud is actually higher in the big, fancy journals, and that many studies that end up in lower-tier journals actually go through *more* peer review in some cases, as they have been rejected from higher impact publications.

Unfortunately as I was moderating, I wasn’t taking notes, and I can’t recall what we ended the session on (but it was a great comment and general agreement that it nicely tied things up). I’ve also tried to Storify the session based on the #medskep hashtag, but I’m new to Storify and it doesn’t want to embed for me. If you were there, please feel free to add to the discussion in the comments below.

Fear & vaccines

I realize that, despite the scientific evidence to the contrary, there is still a lot of fear and misunderstanding about vaccine safety. Two recent articles discuss this “epidemic of fear” and why it affects us all, the first here at Wired magazine, and the second here at the Gotham Skeptic. I especially like the second, which has some excellent points:

My pediatric practice is situated at the nexus of three Manhattan neighborhoods (the West Village, Chelsea, and the Meat Packing District) that seem to comprise just the right balance of wealth, edginess, and socio-cultural awareness that lends itself to this new mistrust of vaccines. But these neighborhoods are not unique. According to sources at the NYC DOH, the Upper West Side of Manhattan and Park Slope in Brooklyn are also hot-spots of parental vaccine resistance. What stands out about these neighborhoods, and others like them, is that they contain a high percentage of middle to upper middle class families that tend to be young, well educated, and liberal in their political and social views. Because I live in one of these areas, work in another, and fit this description pretty squarely, I can identify with the underlying tendencies at work behind the concerns of these parents. A healthy questioning of authority (doctors), an underlying mistrust in the competence of the government (the CDC), overt mistrust and a general level of cynicism of big business (the pharmaceutical industry), and a sense of empowerment that comes with one’s social status, all contribute to this tendency to mistrust vaccines and those who recommend them. The difference between these concerned parents and myself (also a parent), is an understanding of the scientific method and the role it plays in this issue. One term that I have purposefully left out as a key element in this new epidemic is “skepticism.” While many of these parents believe they are being skeptical of vaccines, their manufacturers and the agencies that recommend them, this couldn’t be further from the truth. What they are being is misled and taken advantage of. They would actually be better characterized as anti-skeptics. To quote Brian Dunning of Skeptoid.com:

“The true meaning of the word skepticism has nothing to do with doubt, disbelief, or negativity. Skepticism is the process of applying reason and critical thinking to determine validity. It’s the process of finding a supported conclusion, not the justification of a preconceived conclusion.”