Tag Archives: Guns

Politics, Science and Edumacation

Mother Jones Revisited

A couple of years ago, Mother Jones did a study of mass shootings which attempted to characterize these awful events. Some of their conclusions were robust — such as the finding that most mass shooters acquire their guns legally. However, their big finding — that mass shootings are on the rise — was highly suspect.

Recently, they doubled down on this, proclaiming that Harvard researchers have confirmed their analysis1. The researchers use an interval analysis to look at the time differences between mass shootings and claim that the recent run of short intervals proves that the mass shootings have tripled since 2011.2

Fundamentally, there’s nothing wrong with the article. But practically, there is: they have applied a sophisticated technique to suspect data. This technique does not remove the problems of the original dataset. If anything, it exacerbates them.

As I noted before, the principle problem with Mother Jones’ claim that mass shootings were increasing was the database. It had a small number of incidents and was based on media reports, not by taking a complete data set and paring it down to a consistent sample. Incidents were left out or included based on arbitrary criteria. As a result, there may be mass shootings missing from the data, especially in the pre-internet era. This would bias the results.

And that’s why the interval analysis is problematic. Interval analysis itself is useful. I’ve used it myself on variable stars. But there is one fundamental requirement: you have to have consistent data and you have to account for potential gaps in the data.

Let’s say, for example, that I use interval analysis on my car-manufacturing company to see if we’re slowing down in our production of cars. That’s a good way of figuring out any problems. But I have to account for the days when the plant is closed and no cars are being made. Another example: let’s say I’m measuring the intervals between brightness peaks of a variable star. It will work well … if I account for those times when the telescope isn’t pointed at the star.

Their interval analysis assumes that the data are complete. But I find that suspect given the way the data were collected and the huge gaps and massive dispersion of the early intervals. The early data are all over the place, with gaps as long as 500-800 days. Are we to believe that between 1984 and 1987, a time when violent crime was surging, that there was only one mass shooting? The more recent data are far more consistent with no gap greater than 200 days (and note how the data get really consistent when Mother Jones began tracking these events as they happened, rather than relying on archived media reports).

Note that they also compare this to the average of 172 days. This is the basis of their claim that the rate of mass shootings has “tripled”. But the distribution of gaps is very skewed with a long tail of long intervals. The median gap is 94 days. Using the median would reduce their slew of 14 straight below-average points to 11 below-median points. It would also mean that mass shootings have increased by only 50%. Since 1999, the median is 60 days (and the average 130). Using that would reduce their slew of 14 straight short intervals to four and mean that mass shootings have been basically flat.

The analysis I did two years ago was very simplistic — I looked at victims per year. That approach has its flaws but it has one big strength — it is less likely to be fooled by gaps in the data. Huge awful shootings dominate the number of victims and those are unlikely to have been missed in Mother Jones’ sample.

Here is what you should do if you want to do this study properly. Start with a uniform database of shootings such as those provided by law enforcement agencies. Then go through the incidents, one by one, to see which ones meet your criteria.

In Jesse Walker’s response to Mother Jones, in which he graciously quotes me at length, he notes that a study like this has been done:

The best alternative measurement that I’m aware of comes from Grant Duwe, a criminologist at the Minnesota Department of Corrections. His definition of mass public shootings does not make the various one-time exceptions and other jerry-riggings that Siegel criticizes in the Mother Jones list; he simply keeps track of mass shootings that took place in public and were not a byproduct of some other crime, such as a robbery. And rather than beginning with a search of news accounts, with all the gaps and distortions that entails, he starts with the FBI’s Supplementary Homicide Reports to find out when and where mass killings happened, then looks for news reports to fill in the details. According to Duwe, the annual number of mass public shootings declined from 1999 to 2011, spiked in 2012, then regressed to the mean.

(Walker’s article is one of those “you really should read the whole thing” things.)

This doesn’t really change anything I said two year ago. In 2012, we had an awful spate of mass shootings. But you can’t draw the kind of conclusions Mother Jones wants to from rare and awful incidents. And it really doesn’t matter what analysis technique you use.

1. That these researchers are from Harvard is apparently a big deal to Mother Jones. As one of my colleague used to say, “Well, if Harvard says it, it must be true.”↩

2. This is less alarming than it sounds. Even if we take their analysis at face value, we’re talking about six incidents a year instead of two for a total of about 30 extra deaths or about 0.2% of this country’s murder victims or about the same number of people that are crushed to death by their furniture. We’re also talking about two years of data and a dozen total incidents.↩

Politics

Mother Jones Doesn’t Know Data

You know, you could probably cut out a career in responding to Mother Jones twisting and distorting of data from gun deaths. Today has another wonderful example. Hopping on the rather hysterical claim that gun deaths are close to exceeding traffic deaths, they look at it at a state by state level and conclude that “It’s little surprise that many of these states—including Alaska, Arizona, Colorado, Indiana, Utah, and Virginia—are notorious for lax gun laws.”

Look at the map. Then look at this one which shows the Brady Campaign’s scorecard for state laws on guns. The states were gun deaths exceed traffic deaths are Alaska (Bradley score 0), Washington (48), Oregon (38), California (81!!), Nevada (5), Utah (0), Arizona (0), Colorado (15), Missouri (4), Illinois (35), Louisiana (2), Michigan (25), Ohio (7) and Virginia (12). Of the 14 states, half have Brady scores over 12 and California has the most restrictive gun laws in the nation.

Going by rate of gun ownership, the states are Alaska (3rd highest gun ownership rate in nation), Washington (33), Oregon (28), California (44), Nevada (38), Utah (16), Arizona (32), Colorado (36), Missouri (15), Illinois (43), Louisiana (13), Michigan (27), Ohio (37) and Virginia (35). In other words, the states where traffic deaths exceed gun death are just as likely to have a low gun ownership rate as a high one.

Oops.

Moreover, the entire “guns are killing more than cars” meme is garbage to begin with. Gun deaths, as I have said in every single post on this subject, have fallen over the last twenty years. The thing is that traffic deaths have fallen even faster. The gun grabbers might have had a point back in 1991, when we had a spike in gun deaths that caused them to almost exceed traffic deaths. But they don’t now because both rates are down, way down. Traffic fatalities, in particular, plunged dramatically in the mid-00’s.

A real analysis of the data would look at both factors to see if better drunk driving laws or seatbelt laws or whatever are also playing a factor here. But Mother Jones isn’t interested in that (for the moment). What they are interested in is stoking panic about guns.

(Notice also that MJ illustrates their graph with a picture of an assault rifle, even though these are responsibly for a tiny number of gun deaths.)

Politics

Mother Jones Again. Actually Texas State

Mother Jones, not content with having running one of the more bogus studies on mass shootings (for which they boast about winning an award from Ithaca College), is crowing again about a new study out of Texas State. They claim that the study shows that mass shooting are rising, that available guns are the reason and that civilians never stop shootings.

It’s too bad they didn’t read the paper too carefully. Because it supports none of those conclusions.

  • The Texas State study covers only 84 incidents. Their “trend” is that about half of these incident happened in the last two years of the study. That is, again, an awfully small number to be drawing conclusions from.
  • The data are based on Lexis/Nexus searches. That is not nearly as thorough as James Alan Fox‘s use of FBI crime stats and may measure media coverage more than actual events. They seem to have been reasonably thorough but they confirm their data from … other compilations.
  • Their analysis only covers the years 2000-2010. This conveniently leaves out 2011 (which had few incidents) and the entirety of the 80’s and 90’s, when crime rates were nearly twice what they are now. The word for this is “cherry picking”. Consider what their narrow year range means. If the next decade has fewer incidents, the “trend” becomes a spike. Had you done a similar study covering the years 1990-2000, using MJ’s graph, you would have concluded that mass shootings were rising then. But this would have been followed by five years with very few active shooter events. Look at Mother Jones’ graph again. You can see that mass shootings fell dramatically in the early 2000’s, then spiked up again. That looks like noise in a flat trend over a 30-year baseline. But when you analyze it the way the Blair study does, it looks like a trend. You know what this reminds me of? The bad version of global warming skepticism. Global warming “skeptics” will often show temperature graphs that start in 1998 (an unusually warm year) and go the present to claim that there is no global warming. But if you look at the data for the last century, the long-term trend becomes readily apparent. As James Alan Fox has show, the long-term trend is flat. What Mother Jones has done is jump on a study that really wasn’t intended to look at long-term trends and claim it confirms long-term trends.
  • Mother Jones’ says: “The unprecedented spike in these shootings came during the same four-year period, from 2009-12, that saw a wave of nearly 100 state laws making it easier to obtain, carry, and conceal firearms.” They ignore that the wave of gun law liberalization began in the 90’s, before the time span of this study.
  • MJ also notes that only three of the 84 attacks were stopped by the victims using guns. Ignored in their smugness is that a) that’s three times what Mother Jones earlier claimed over a much longer time baseline; b) the number of incidents stopped by the victims was actually 16. Only three used guns.; c) at least 1/3 of the incident happened in schools, were guns are forbidden.

    • So, yeah. They’re still playing with tiny numbers and tiny ranges of data to draw unsupportable conclusions. To be fair, the authors of the study are a bit more circumspect in their analysis, which is focused on training for law enforcement in dealing with active shooter situations. But Mother Jones never feels under any compulsion to question their conclusions.

    (H/T: Christopher Mason)

    Update: You might wonder why I’m on about this subject. The reason is that I think almost any analysis of mass shootings is deliberately misleading. Over the last twenty years, gun homicides have declined 40% (PDF) and gun violence by 70%. This is the real data. This is what we should be paying attention to. By diverting our attention to these horrific mass killings, Mother Jones and their ilk are focusing on about one one thousandth of the problem of gun violence because that’s the only way they can make it seem that we are in imminent danger.

    The thing is, Mother Jones does acknowledge the decline in violence in other contexts, such as claiming that the crackdown on lead has been responsible for the decline in violence. So when it suits them, they’ll freely acknowledge that violent crime has plunged. But when it comes to gun control, they pick a tiny sliver of gun violence to try to pretend that it’s not. And the tell, as I noted before, is that in their gun-control articles, they do not acknowledge the overall decline of violence.

    Using a fact when it suits your purposes and ignoring it when it doesn’t is pretty much the definition of hackery.

    Politics

    Mathematical Malpractice: Spree Killings Again

    1 Comment

    This analysis, which claims that the US has more school spree killings than 36 nations combined, is getting a lot of play. It shouldn’t. It is extremely bad mathematical malpractice.

    The basic reason it is mathematical malpractice is the same reason the Mother Jones study was: it is difficult to analyze extremely rare events. When you narrow your investigation to events that happen maybe once a decade and are compiled haphazardly, you are simply going to be dominated by small number statistics and selection bias. You can therefore use those numbers to say, basically, anything you want.

    Let’s break down just how bad the numbers are being twisted here.

    1) The sample ends in 2009. That excludes the recent spate of knife attacks in Chinese schools that have left 21 dead. If you did this analysis a week ago, you would have had to drop China from the right column.

    2) The sample excludes acts of terror or war. But if Islamists shoot up a school because they don’t want girls to read, are those kids any less dead? If a drone strike misses its targets and kills a classroom, are those kids less dead? Why must we exclude the Beslan attack that left 186 kids dead?

    3) The sample excludes single homicides, which amount to 302 deaths in the United States over the time involved and God knows how many in other countries. So you are literally excluding 90% of the problem and focusing just on a tiny subset of killings.

    4) Comparing us to 36 other countries is ridiculous when some of those countries are places like Bosnia-Herzegovina (population 4 million). We have more population, period, then 30 of the countries on that list combined. Also included in that list of countries are England, Scotland and Northern Ireland, which are, technically speaking, not countries.

    5) The problem of small number statistics can be best illustrated by playing with the data a bit. If I include the knife attacks and move China into the left column, suddenly China has more violent deaths than 30 other countries. If I move Germany onto the left side, suddenly they have more spree killings than a bunch of other countries. If I define the sample in the 1990’s, suddenly Australia dominates the statistics. You simple can not draw conclusions from samples that are that sensitive to single events.

    6) Combining points 4 and 5, if you look at spree killing rates rather than the deliberate mathematical malpractice of comparing absolute numbers, the situation is very different. In 2009, the Winnenden shooting killed 15. Scaled up to the population of the United States, that would be the equivalent of 60 people dead, more than the worst year the United States has ever had. In 1996, 35 people were kill in Port Arthur, Australia. Scaled up to the US population, that would almost equal 500 dead. It is an event that is seared into the memories of Australians. My point is not that these countries are worse than we are. My point is that these are rare and horrible events and you can manipulate the numbers to prove anything you want.

    7) The biggest thing missing here is a sense of time. Is the rate of school killings going up or down? The answer, of course, is down. Check out chart one at Ezra Klein’s blog that shows that the rate of assault death has fallen by over half since 1970. Check out the NCES page I link above which shows a significant decline in on-campus homicides, from 40/year in the 90’s to 30/year in the 00’s. That decline is a hundred more kids running in the sunshine. The NCES data, based on a complete sample of over 600 incidents, is useful. This …. isn’t.

    I’m not trying to downplay the horror that unfolded on Friday. However, I don’t think any debate can proceed unless we have a good grasp of the problem we are trying to solve. Far too many children are murdered in school in this country — that was as true on Thursday as it is today. But to be useful, the debate needs to be on honest terms. Committing mathematical malpractrice by deceptively comparing the United States to 36 other countries as though there something to be learned from that is not an honest debate and is likely to produce a panicky and ill-considered response.