Sunday, December 28, 2008

Reuters does social science, badly

This article from Reuters, forwarded to me by one of the students in my just-completed undergraduate econometrics course, nicely illustrates everything I complain about in that course regarding discussions of social science in the supposedly highbrow media.

First, we learn:
The survey examined sexual experiences as well as family relationships and lifestyle habits of Japanese females and males aged 16 to 49. It was carried out in September and was based on about 1,500 people.
This is all well and good but on might have a few tiny questions still remaining about the data collection. For example, was it a random sample of the Japanese population or was it a sample of, say, subscribers to a magazine? Second, what was the response rate? Was it something reasonable, like 80 percent, or something awful, like 20 percent?

Next we learn that:
The average age of first-time sex for those who said they ate breakfast every day as a middle school student was 19.4, while for those who skipped breakfast, the average age was 17.5.

Did they really just take a mean difference and report it? Perhaps there were some conditioning variables, though the text sounds like there were not. Perhaps they did some sort of highly advanced statistical procedure, like a t-test, to gain some sense of how likely the observed difference is to arise by chance in a world where this is in fact no difference in population proportions? Is there any conceivable reason why the reader should suspect that these estimates represent causal effects?

But it appears that the authors of this study had no time to waste on foolish technical trivia. There are important interpretations to be made, such as this:

"The fact that people can't eat breakfast may show something about their family environment," said Kitamura. "Before blaming individuals for having sex at an early age, it may be necessary to look into the sort of homes they are from."

There is even a hint of some policy conclusions and, of course, more things for our friends at the government to do when they aren't busy doing such a bang-up job of looking after the economy.

What we do not learn, in addition to all these details, is where to find the study so that we can read it ourselves. Would it be too much to provide a link? Isn't that the very start of using the wonderful power of the web to create new knowledge?

Absent a link, perhaps the curious reader could learn the name of the journal in which the study was published. Oh, wait, the study is not published yet. Reuters is telling the world about a study that has not even been through peer review. Sigh.

Score a big fat F for Reuters on this one and another big fat F for the Japan Family Planning Association.

Hat tip: Russell Bittmann