# Mathematical Malpractice Watch: A Trilogy of Error

Three rather ugly instances of mathematical malpractice have caught my attention in the last month. Let’s check them out.

The Death of Facebook or How to Have Fun With Out of Sample Data

Last month, Princeton researchers came out with the rather spectacular claim that the social network Facebook would be basically dead within a few years. The quick version is that they fit an epidemiological model to the rise and fall of MySpace. They then used that same model, varying the parameters, to fit Google trends on searches for Facebook. They concluded that Facebook would lose 80% of its customers by 2017.

This was obviously nonsese as detailed here and here. It suffered from many flaws, notably assuming that the rise and fall of MySpace was necessarily a model for all social networks and the dubious method of using Google searches instead of publicly available traffic data as their metric.

But there was a deeper flaw. The authors fit a model of a sharp rise and fall. They then proclaim that this model works because Facebook’s google data follows the first half of that trend and a little bit of the second. But while the decline in Facebook Google searches is consistent with their model, it is also consistent with hundreds of others. It would be perfectly consistent with a model that predicts a sharp rise and then a leveling off as the social network saturates. Their data are consistent with but not discriminating against just about any model.

The critical part of the data — the predicted sharp fall in Facebook traffic — is out of sample (meaning it hasn’t happened yet). But based on a tiny sliver of data, they have drawn a gigantic conclusion. It’s Mark Twain and the length of the Mississippi River all over again.

We see this a lot in science, unfortunately. Global warming models often predict very sharp rises in temperature — out of sample. Models of the stock market predict crashes or runs — out of sample. Sports twerps put together models that predict Derek Jeter will get 4000 hits — out of sample.

Anyone who does data fitting for a living knows this danger. The other day, I fit a light curve to a variable star. Because of an odd intersection of Fourier parameters, the model predicted a huge rise in brightness in the middle of its decay phase because there were no data to constrain it there. So it fit a small uptick in the decay phase as though it were the small beginning of a massive re-brightening.

The more complicated the model, the more danger there is of drawing massive conclusions from tiny amounts of data or small trends. If the model is anything other than a straight line, be very very wary at out-of-sample predictions, especially when they are predicting order-of-magnitude changes.

A Rape Epidemic or How to Reframe Data:

The CDC recently released a study that claimed that 1.3 million women were raped and 12.6 million more were subject to sexual violence in 2010. This is six or more times the estimates of the FBI’s extremely rigorous NCVS estimate. Christina Hoff Summers has a breakdown of why the number is so massive:

It found them by defining sexual violence in impossibly elastic ways and then letting the surveyors, rather than subjects, determine what counted as an assault. Consider: In a telephone survey with a 30 percent response rate, interviewers did not ask participants whether they had been raped. Instead of such straightforward questions, the CDC researchers described a series of sexual encounters and then they determined whether the responses indicated sexual violation. A sample of 9,086 women was asked, for example, “When you were drunk, high, drugged, or passed out and unable to consent, how many people ever had vaginal sex with you?” A majority of the 1.3 million women (61.5 percent) the CDC projected as rape victims in 2010 experienced this sort of “alcohol or drug facilitated penetration.”

What does that mean? If a woman was unconscious or severely incapacitated, everyone would call it rape. But what about sex while inebriated? Few people would say that intoxicated sex alone constitutes rape — indeed, a nontrivial percentage of all customary sexual intercourse, including marital intercourse, probably falls under that definition (and is therefore criminal according to the CDC).

Other survey questions were equally ambiguous. Participants were asked if they had ever had sex because someone pressured them by “telling you lies, making promises about the future they knew were untrue?” All affirmative answers were counted as “sexual violence.” Anyone who consented to sex because a suitor wore her or him down by “repeatedly asking” or “showing they were unhappy” was similarly classified as a victim of violence. The CDC effectively set a stage where each step of physical intimacy required a notarized testament of sober consent.

In short, they did what is called “reframing”. They took someone’s experiences, threw away that person’s definition of them and substituted their own definition.

This isn’t the first time this has happened with rape stats nor the first time Summers had uncovered this sort of reframing. Here is an account of how researchers decided that women who didn’t think they had been raped were, in fact, raped, so they could claim a victimization rate of one in four.

Scientists have to classify things all the time based on a variety of criteria. The universe is a messy continuum; to understand it, we have to sort things into boxes. I classify stars for a living based on certain characteristics. The problem with doing that here is that women are not inanimate objects. Nor are they lab animals. They can have opinions of their own about what happened to them.

I understand that some victims may reframe their experiences to try to lessen the trauma of what happened to them. I understand that a woman can be raped but convince herself it was a misunderstanding or that it was somehow her fault. But to a priori reframe any woman’s experience is to treat them like lab rats, not human beings capable of making judgements of their own.

But it also illustrates a mathematical malpractice problem: changing definitions. This is how 10,000 underage prostitutes in the United States becomes 200,000 girls “at risk”. This is how small changes in drug use stats become an “epidemic”. If you dig deep into the studies, you will find the truth. But the banner headline — the one the media talk about — is hopelessly and deliberately muddled.

Sometimes you have to change definitions. The FBI changed their NCVS methodology a few years ago on rape statistics and saw a significant increase in their estimates. But it’s one thing to hone; it’s another to completely redefine.

(The CDC, as my friend Kevin Wilson pointed out, mostly does outstanding work. But they have a tendency to jump with both feet into moral panics. In this case, it’s the current debate about rape culture. Ten years ago, it was obesity. They put out a deeply flawed study that overestimated obesity deaths by a factor of 14. They quickly admitted their screwup but … guess which number has been quoted for the last decade on obesity policy?)

You might ask why I’m on about this. Surely any number of rapes is too many. The reason I wanted to talk about this, apart from my hatred of bogus studies, is that data influences policy. If you claim that 1.3 million women are being raped every year, that’s going to result in a set of policy decisions that are likely to be very damaging and do very little to address the real problem.

If you want a stat that means something, try this one: the incidence of sexual violence has fallen 85% over the last 30 years. That is from the FBI’s NCVS data so even if they are over- or under-estimating the amount of sexual violence, the differential is meaningful. That data tells you something useful: that whatever we are doing to fight rape culture, it is working. Greater awareness, pushing back against blaming the victim, changes to federal and state laws, changes to the emphasis of attorneys general’s offices and the rise of internet pornography have all been cited as contributors to this trend.

That’s why it’s important to push back against bogus stats on rape. Because they conceal the most important stat; the one that is the most useful guide for future policy and points the way toward ending rape culture.

The Pending Crash or How to Play with Scales:

Yesterday morning, I saw a chart claiming that the recent stock market trends are an eerie parallel of the run-up to the 1929 crash. I was immediately suspicious because, even if the data were accurate, we see this sort of crap all the time. There are a million people who have made a million bucks on Wall Street claiming to pattern match trends in the stock market. They make huge predictions, just like the Facebook study above. And those predictions are always wrong. Because, again, the out of sample data contains the real leverage.

This graph is even worse than that, though. As Quartz points out, the graph makers used two different y-axes. In one, the the 1928-29 rise of the stock market was a near doubling. In the other, the 2013-4 rise was an increase of about 25%. When you scale them appropriately, the similarity vanishes. Or, alternatively, the pending “crash” would be just an erasure of that 25% gain.

I’ve seen this quite a bit and it’s beginning to annoy me. Zoomed-in graphs of narrow ranges of the y-axis are used to draw dramatic conclusions about … whatever you want. This week, it’s the stock market. Next week, it’s global warming skeptics looking at little spikes on a 10-year temperature plot instead of big trends on a 150-year one. The week after, it will be inequality data. Here is one from Piketty and Saez, which tracks wealth gains for the rich against everyone else. Their conclusion might be accurate but the plot is useless because it is scaled to intervals of \$5 million. So even if the bottom 90% were doing better, even if their income was doubling, it wouldn’t show up on the graph.

# Lonely Among Us

I’m a little late on this, but the Atlantic ran a recent story arguing that all of our social networking is making us lonelier than ever. There are a few leaps of logic that are too much to me, such as the leadoff anecdote about the lonely death of Yvette Vickers. The author regards it as somehow horrifying that she died alone, unnoticed for six months and her only communications had been with old fans.

Why is that a problematic anecdote? Because it’s not like people have never died alone before. What was unique about this case was not that a woman died alone and no one noticed for a while. She was childless, not religious and most of her friends were dead. What was unique was that she was not alone; that her contacts with distant fans, however superficial, at least brought her some flitting human contact.

The article maintains an early balance, pointing out the social media mainly amplify our existing social structure and it does not appear that social media are causing the rise in loneliness. But then it goes off onto one of the most aggravating journalistic excess: the personal stream of consciousness. It mainly rehashes the same argument we have been hearing for years: social media create an artificial social image, social media are superficial, etc., etc.

The problem is that the Facebook experience she describes is atypical. There are narcissistic people out there who have a zillion friends and carefully cultivate their image. But for almost all of us, it’s a way to stay in contact with people we actually know, to dump off a quick update in the busy world to let people know what’s going on. The typical user has about 130 friends, which is close to what our brains can deal with. And they know most of them pretty well.

For many, social media are not a replacement for social contact but an intensifier of it. I mentioned last week how I used Facebook to alert everyone I knew to be gallbladder problem. In the process, I heard from several people about their own gallbladder surgeries. Maybe, in the pre-internet days, they would have called the hospital to talk to me. Maybe. But I doubt it.

Facebook allows me to send pictures to my parents and keep them up to date on their granddaughter. The last time we were in Australia, it allowed my wife to meet up with a childhood friend for the first time in decades. I have had numerous good conversations start from, “Hey, I saw what you said on Facebook yesterday.”

My political blogging fits the loneliness description more. But while it’s true that the blog and twitter feed don’t harvest close personal friends (and probably does feed some narcissism), it does give me an outlet for stuff I’d just be pacing the room and ranting about. It does, hopefully, give some of my readers something to talk about to their friends. And it allows my friends to choose whether they want to deal with my politics.

Sullivan’s readers pushed back hard on this, pointing out actual research that shows that an internet user is less isolated than a non-internet user in the same circumstance. Think of how awful it would have been for Yvette Vickers without the fleeting contact of the internet.

In the end, have heard this line of crap since the dawn of time. Every invention from the printing press to e-mail was supposed to make us a soulless society, to deprive us of real human contact. I’m sure, when man first painted figures on the walls of caves, some self-important dick was saying, “Well, this is all fine. But we’re becoming a soulless society. People don’t pantomime buffalo hunts anymore.”

But it seems, as the article argues in its more sensible paragraphs, that this is something we have chosen: to have a world that is more connected than ever even as we get lonelier for various reasons that are probably completely unrelated to internet technology. The decline in families and tight-knit communities is a loss. But we are also in world where someone is seconds away from communicating their thoughts to millions, where friendships can be forged over almost anything and where one needs never lose contact with old friends. I too am concerned about the reconfiguring of our social model. But I’m unwilling to get hysterical about it.

Humans are social animals, no matter what the misanthropy people might think. We will never move to a society where people prefer loneliness over companionship or machines over people (a few genetically self-correcting exceptions aside). I see the enthusiasm for social media as a response to loneliness, not a cause of it. And as such, it’s a good thing.

Update: More from Althouse.