Posts Tagged ‘Mathematical Malpractice’

Mother Jones Hacks Again

Monday, February 18th, 2013

A few weeks ago Mother Jones, having not learned the lesson of their absurd article claiming mass shootings are on the rise, published a list of 10 Myths about guns and gun control from Dave Gilson. And I’m going to debunk their debunking again because the article represents what I believe is one of the worst sins in the field of Mathematical Malpractice: cherry-picking. As I went through this, it became obvious that MJ was not interested in the facts, really. What was motivating them was the argument. And so they picked any study — no matter how small, how biased or how old — to support their point. They frequently ignore obvious objections and biases. And they sometimes ignore larger more detailed studies in favor of the smaller ones if it will support their contention.

We see this a lot in the punditocracy, unfortunately. As Bill James said, most people use studies the way a drunk uses a lamppost — for support, not illumination. In any sufficiently advanced but difficult field of study, you will find multiple studies examining an issue. Let’s say it’s a supposed connection between watching Glee and having a heart attack. If there is, in reality, no connection between the two, you might find eight studies that show no connection, one that shows an anti-correlation and one that shows a correlation. This is fine. This is science. There are always outlier studies even if all the researchers are completely ethical and honest. The outliers fall away when your interest is the question and you look at all the evidence. But the outliers dominate the discussion from those who have an agenda.

This happens a lot in the gun debate. On both sides, really. But Mother Jones’ article is a particularly putrid example of this because that’s basically all it does: collect the cherry-picked nonsensical studies that support their anti-gun agenda. It’s quite remarkable actually; almost a clinic in how not to do research.

But here’s the one thing that really tips you off. There is one myth that Mother Jones does not debunk. It’s a myth that’s really independent of what you think of gun ownership … unless you’ve already staked part of your reputation and agenda on the myth that gun violence is increasing. In fact, all forms of violent crime have been falling for twenty years. This is, in my mind, the single most important fact in debates over crime and violence and the single most important myth to debunk.

MJ does not address this myth. They don’t even talk about it. That is a huge tell.

(more…)

Tags: , ,
Posted in Civil Liberties, Politics | 3 Comments »

Mathematical Malpractice: Spree Killings Again

Monday, December 17th, 2012

This analysis, which claims that the US has more school spree killings than 36 nations combined, is getting a lot of play. It shouldn’t. It is extremely bad mathematical malpractice.

The basic reason it is mathematical malpractice is the same reason the Mother Jones study was: it is difficult to analyze extremely rare events. When you narrow your investigation to events that happen maybe once a decade and are compiled haphazardly, you are simply going to be dominated by small number statistics and selection bias. You can therefore use those numbers to say, basically, anything you want.

Let’s break down just how bad the numbers are being twisted here.

1) The sample ends in 2009. That excludes the recent spate of knife attacks in Chinese schools that have left 21 dead. If you did this analysis a week ago, you would have had to drop China from the right column.

2) The sample excludes acts of terror or war. But if Islamists shoot up a school because they don’t want girls to read, are those kids any less dead? If a drone strike misses its targets and kills a classroom, are those kids less dead? Why must we exclude the Beslan attack that left 186 kids dead?

3) The sample excludes single homicides, which amount to 302 deaths in the United States over the time involved and God knows how many in other countries. So you are literally excluding 90% of the problem and focusing just on a tiny subset of killings.

4) Comparing us to 36 other countries is ridiculous when some of those countries are places like Bosnia-Herzegovina (population 4 million). We have more population, period, then 30 of the countries on that list combined. Also included in that list of countries are England, Scotland and Northern Ireland, which are, technically speaking, not countries.

5) The problem of small number statistics can be best illustrated by playing with the data a bit. If I include the knife attacks and move China into the left column, suddenly China has more violent deaths than 30 other countries. If I move Germany onto the left side, suddenly they have more spree killings than a bunch of other countries. If I define the sample in the 1990′s, suddenly Australia dominates the statistics. You simple can not draw conclusions from samples that are that sensitive to single events.

6) Combining points 4 and 5, if you look at spree killing rates rather than the deliberate mathematical malpractice of comparing absolute numbers, the situation is very different. In 2009, the Winnenden shooting killed 15. Scaled up to the population of the United States, that would be the equivalent of 60 people dead, more than the worst year the United States has ever had. In 1996, 35 people were kill in Port Arthur, Australia. Scaled up to the US population, that would almost equal 500 dead. It is an event that is seared into the memories of Australians. My point is not that these countries are worse than we are. My point is that these are rare and horrible events and you can manipulate the numbers to prove anything you want.

7) The biggest thing missing here is a sense of time. Is the rate of school killings going up or down? The answer, of course, is down. Check out chart one at Ezra Klein’s blog that shows that the rate of assault death has fallen by over half since 1970. Check out the NCES page I link above which shows a significant decline in on-campus homicides, from 40/year in the 90′s to 30/year in the 00′s. That decline is a hundred more kids running in the sunshine. The NCES data, based on a complete sample of over 600 incidents, is useful. This …. isn’t.

I’m not trying to downplay the horror that unfolded on Friday. However, I don’t think any debate can proceed unless we have a good grasp of the problem we are trying to solve. Far too many children are murdered in school in this country — that was as true on Thursday as it is today. But to be useful, the debate needs to be on honest terms. Committing mathematical malpractrice by deceptively comparing the United States to 36 other countries as though there something to be learned from that is not an honest debate and is likely to produce a panicky and ill-considered response.

Tags: , , ,
Posted in Politics | 1 Comment »

Tuesday Linkorama

Wednesday, November 28th, 2012
  • An interesting article on how child abuse panic is keeping men out of the childcare industry. My daughter had a male teacher at her school for a while. She really liked him and it was good to see her having a male role model in her life beyond me. But I also admired the man’s courage; I would not put myself in such a vulnerable position.
  • One of my favorite things to do as a grad student was to look up heavily referenced papers to see if they said what people said they said. At least a quarter of the time, they didn’t. Maggie McNeill just dug up a 30-year-old bit of Mathematical Malpractice that’s been cited incorrectly in support of innumerable bad laws.
  • A frustrating story about why we can’t watch WKRP in its original format. We really have to do something about fair use. The Republicans indicated that they might; then ran away from that position.
  • This video, of a hilarious bug in the FIFA 2012 video game, had me giggling.
  • I have to disagree with almost everything in this article claiming the alcohol industry is trying to make us drunks. It assumes alcoholism is entirely a function of government policy. And it mainly reads like a press release from the powerful forces trying to overturn the SCOTUS decision on out-of-state liquor importation, an issue of particular relevance to Pennsylvania.
  • Is airport security taking more lives than it is saving? Seems like.
  • I’ve been sitting on this story, about how doctor witheld information about a child’s medical future from the parents, for a while, trying to think of a way to approach it. Might still write a long form post. But I default to thinking people have a right to know. To presume to make that decision for them is arrogance. As our diagnostic tools get better, we need to give people the legal option: do you want know if we find anything bad? What happens if a cure is invented and this kid doesn’t know that he needs one?

    • Tags: , , , , , , , , ,
    Posted in 'Culture', Healthcare, Humor, The Nanny State, War on Terror | Comments Off

    Mathematical Malpractice Watch: Marginal Taxes

    Wednesday, November 21st, 2012

    I’ve noticed a little flaw in commentary lately. The Left Wing, in their push to raise taxes, are citing work like this, which claims the Laffer Curve peaks at 50-70% and therefore we could massively raise taxes on the wealthy.

    Let’s put aside that you never want government taxation rates to be at the peak of the Laffer Curve (it leaves you zero fiscal room for emergencies and means you’re crippling the economy but not quite enough to depress tax revenue). The problem is that we already pretty close to that peak. For the very wealthy, the marginal income tax rate is 35%. Medicare tax is another 2.9% (employer+employee). We’ll ignore Social Security tax under the assumption we’re just dealing with millionaires. Then you have state income taxes, which range from 0% in states like Texas to a top marginal rate of 11% in Oregon and Hawaii. So marginal tax rates are currently at 38-49%, which is pretty much the lower bound of what the rather optimistic Diamond and Saez say is the peak of the Laffer Curve. And since it’s a Laffer Curve, not the Laffer Triangle, it starts bending before it rolls over, so we’re probably getting within shouting distance of peak revenue already.

    I’m not saying whether we should or should not raise taxes (I’ve come out on the other blog in favor of raising them on everyone since I see little alternative given our present circumstances). But let’s at least debate honestly about where we are on marginal rates, huh?

    Tags: ,
    Posted in Taxes and Spending | Comments Off

    Mathematical Malpractice Watch: Weather Fatalities

    Tuesday, November 20th, 2012

    This came to my attention a month ago. I drafted a post, forgot about it in the election/migraine event horizon but now want to get it out my drafts section. I think it’s worth posting because we are likely to hear more of this from the more hysterical environmental wing.

    The chart, from Ezra Klein’s usually excellent Wonkblog, purports to show a steep rise in weather-related fatalities in recent years.

    It doesn’t show anything of the kind.

    First of all, what it shows is a slight decline or flat trend with a few recent spikes caused by a 90′s heat wave, Hurricane Katrina and last year’s tornados. Now maybe you can argue that we should pay more attention to these in the era of global warming because they may be related (or may not). I agree. However, the long term trend in almost all categories is down — way down. Deaths from lightning strikes are down by over two-thirds over the last 70 years. That’s real progress.

    But the progress is even better than the graph shows. The graph makes a huge blindingly obvious error; one that Klein’s readers jumped on immediately: it does not account for population growth. The first data point is from a sample of 140 million people while the last if from a sample of 310 million. To compare raw figures is simply ridiculous (and, indeed, Klein’s co-blogger later tweeted a version with death rates that was far less dire and showed dramatic declines in weather-related fatalities).

    The third problem is less obvious but potentially the worst one. The plot includes deaths from heat, cold, “winter fatalities”, rip currents and wind. Heat deaths are particularly important to the point Wonkblog is making since, presumably, global warming will result in more deaths from heat waves and drought.

    The problem is that the NOAA, from whose data the graph is taken, did not track heat deaths until 1986. The same goes for many deaths in the “other” category. Cold fatalities were not tracked until 1988. Winter fatalities until 1986. Rip currents until 2002. Wind deaths until 1995. No correction, none whatsover, is made for the incomplete data that spans the first five or six decades of NOAA’s sample.

    It is simply not sensible to treat the data as though there were zero deaths from heat and other categories before the mid-1980′s. In fact, there are many reasons — the spread of air-conditioning for example — to suspect that heat-related deaths were much much higher in the past. It would defy common sense for the sharp reductions in fatalities from tornados, hurricanes and lightning (not to mention earthquakes) to not reflected in the statistics for other weather-related deaths.

    But let’s not assume. Let’s go to the record. The data start in 1940, which usefully omits one of the greatest environmental calamities in American history: the Dust Bowl. Thousands died; at least 5000 in one 1936 heat wave alone. Another massive drought hit in the 1950′s. A 1972 heat wave killed 900 people. A 1980 heat wave killed 1700 people. All of those happened before the NOAA tracked the number of heat-related deaths. None are in the sample.

    To be completely honest, the NOAA data seems a poor resource for this kind of study. It apparently does not include the 1988 drought, recording only 47 heat-related deaths in that two-year period. But it does include the 1995 and 1999 heat waves. I have no idea what their criteria are. I suspect they are counting deaths from specific short-term heat waves rather than broad massive events like the 88-89 drought. That’s fine as far as it goes. But if your attempt to quantify long-term trends in weather-related deaths ignores droughts; if it ignores the God-damned Dust Bowl, I would submit that you are looking at the wrong data.

    So, in the end, the claim that we are getting more weather-related fatalities than ever is, at least in this case, based on a heavily biased poorly understood sample that barely supports the conclusion

    Tags: ,
    Posted in Science and Edumacation | Comments Off

    Mathematical Malpractice Watch: Guns

    Saturday, September 29th, 2012

    A few weeks ago, Mother Jones did a timeline of mass shootings in response to the spate of summer shootings. The defined their criteria, listed 61 incidents and pointed out, correctly, that most of them were committed with legal firearms.

    The highlight is a map of mass shootings over the last thirty years. The map has some resemblance to Radley Balko’s famous map of botched law enforcement raids. But the use of a map and dots is where the resemblance ends. Balko was very clear that his list of incidents was not, in any way, definitive. And he did not try to parse his incomplete data to draw sketchy conclusions.

    Mother Jones felt under no such compulsion.

    This week, they’ve published an “analysis” of their data and drawn the conclusion that our society has more guns than ever and, perhaps related, more mass shootings. Below, I’ll detail why I think their “analysis” — and yes, I will keep using quotation marks for this — is useless, uninformative and flat-out wrong.

    (more…)

    Tags: , , ,
    Posted in Politics | 2 Comments »

    Mathematical Malpractice Watch: Rape Statistics

    Wednesday, August 22nd, 2012

    The Akin controversy has brought a lot of ugliness to light. I’ll push aside the broader political issues, which I’ve discussed on the other blog. Instead, I want to focus on a particular piece of nonsense from one of the more influential pro-Life figures, who has tried to argue that rape pregnancies are very rare.

    You would think this would be straight forward. You could, for example, find rape victims and ask them if they got pregnant. And indeed, a study estimated that about 5% of rapes end in pregnancy and 32,000 fetuses are conceived in rape every year. You could also ask women getting abortions if they were raped. And indeed, Guttmacher did this and produced a lower estimate of about 10,000 or so. There is no real incentive, at this point, for women to lie about how they got pregnant. Or there are just as many reasons for them lie either way since many women are tragically ashamed of having been raped. But there is no a priori reason to suspect that either number is greatly inflated.

    (The Guttmacher number seems a little better since the numbers in the former study would indicate about 640,000 rapes a year or three times what the FBI and NCVS conclude, even including the under-reporting factor. The difference is likely that Guttmacher is recent and Holmes was published in 1996, shortly after crime rates peaked. Rape rates have plunged in the last 16 years and are down about 85% over the last 30.)

    But plain facts aren’t enough for some people, so John Wilke, “M. D.” has come up with a bizarre rape equivalent of the Drake equation to try to figure out how many rape babies are conceived every year and come up with a vanishingly small number.

    It’s fairly easy to tear apart.

    (more…)

    Tags: , ,
    Posted in Politics | 1 Comment »

    Mathematical Malpractice Watch: Admiration Edition

    Saturday, December 31st, 2011

    Gallup and USA/Today need to be beaten with sticks and sent back to remedial stats class. It’s yet another year and yet another useless poll about the most admired people in America.

    President Obama and Hillary Rodham Clinton are the nation’s most-admired man and woman — again — in the annual USA TODAY/Gallup Poll.

    Each leads their category with 17% of votes in top 10 lists that favor the most familiar names in global politics, religion, entertainment and culture.

    Does this strike you as being meaningful? For 83% of Americans, Barack Obama is not the most-admired man. The #5 man in the poll is Warren Buffett with … 2% of the vote.

    USA Today and Gallup have run this piece of crap for half a century, using an unconstrained plurality. As such, it always puts whoever is most famous — usually the President — at the top of the list. Even in years where George W. Bush would’ve lost an election for Republican dog-catcher, he was still the most admired man because no one else was as famous.

    Unconstrained pluralities are the most useless poll imaginable. They always produce embarrassing or useless results like this. We get this crap with “what issue is most important to you” polls as well, where they’ll say the second most important issue is, say, abortion, when abortion gets about 6% of the vote.

    A more valid way would be to give people a list of potentially admirable people and ask if they admire them. That would be useful and unbiased. Gallup wants to keep it “unprompted”. Fine. Conduct an initial poll, then make the list from the top 100 names on it.

    Polls are created news. They don’t tell us anything and are used to lead a slow news day. But this poll is especially stupid. And yes, I said it when Bush topped the poll too.

    Tags: , ,
    Posted in Science and Edumacation | 1 Comment »

    Mathematical Malpractice Watch: 10 Billion

    Friday, May 6th, 2011

    Seriously? You think we can project population growth a century in advance? Really? You have Nigeria quintupling in population over the next century. That’s predicting the reproductive habits of people who’s great-grandparents have yet to be born.

    What a load of crap.

    Social scientists simply never learn. Every single population projection we have seen for the last fifty years has been too high. Why should we trust them now?

    Tags: ,
    Posted in Politics | Comments Off

    Mathematical Malpractice Watch: Why NationMaster Sucks

    Thursday, April 21st, 2011

    Graphjam ran a graphic today apparently showing all the awful things the US leads the world in.

    It’s crap. It’s clearly produced by someone who spent a few minutes browing nationmaster.com. Nationmaster is convenient but their accuracy is, at best, suspect. There is no uniformity of data and many of the samples are incomplete or old. To be honest, you’re better off going to wikipedia. Much better off.

    But beyond that, they just haven’t thought too much. For example, the graphic has has the US as #1 in crime. This is true, but only because we are a large country and a transparent one. The UK has half as many crimes but a fifth of our population. Germany half as many crimes but a quarter of our population. The crime rate in the US is high but not tops. Same goes with rape, which they have as #1. Scandinavian countries lead the civilized world in that (although likely because they measure their rape stats differently).

    But a lot of this is the nationmaster problem. They have the US as #1 in CO2 emissions. This is actually wrong as China is #1. US emissions have actually been flat over the last few decades. The nationmaster data are 10 years old — way too far out of date. They also have the US as #1 in divorce rate. This is wrong. Russia is #1.

    Teen birth rate? The US is #1 among developed nations. But you have to exclude almost every developing nation in the world to get that ranking. Nationmaster’s data is selective and based on 1994 data. The teen birth rate has plunged since then.

    Heart attacks? I haven’t the faintest clue what they’re showing here. But heart attack survival rates have been growing massively in the US.

    We do lead the world in McDonald’s restaurants and plastic surgery. That tends to come from being the richest country on Earth. We also, unfortunately, lead the world in both prison population and incarceration rate — yet another wonderful effect of our stupid war on drugs.

    Tags: , , ,
    Posted in Politics | Comments Off

    Mathematical Malpractice Watch: Torture Edition

    Thursday, April 14th, 2011

    You know, I’d drafted an entire post about why the Red Cross’ poll showing that young people support torture was bullshit. And then one of Sully’s readers beat me to it. The only thing he left out is that RC poll has no longitudinal information to support the conclusions many commenters have thrown at it. That is, it has no read on whether support for torture has waxed or waned to back up the contention that some sort of post-9/11 PTSD is causing young people to support torture today.

    I would add that the results is unsurprising. Being pro-torture is instinctive to human beings. Indeed, that’s why we place such generous boundaries around acceptable behavior — to avoid people falling into the bottomless dark well that lies in all human hearts. Being anti-torture is a much more difficult position to get to. It requires knowing how intelligence work is done in real life — as opposed to how it is done in movies. It requires a more complex morality than hurting those who’ve hurt you. Those attitudes are not created by age and life experience, but are often strengthened by them. It’s why soldiers and intelligence experts oppose torture.

    Honestly, I’d be surprised if a poll showed anything else.

    Tags: , ,
    Posted in Politics, War on Terror | Comments Off