Alan Schwarz (@alanschwarz) is a Pulitzer Prize-nominated journalist best known for his reportage of public health issues for The New York Times. His 130-article series on concussions in sports is roundly credited with exposing the seriousness of head injuries in the National Football League and all youth athletics. His work was profiled in several films and was a finalist for the 2011 Pulitzer Prize for Public Service, journalism's highest honor. A subsequent series on A.D.H.D. and other psychiatric disorders in children led to his writing " A.D.H.D. NATION: Children, Doctors, Big Pharma and the Making of an American Epidemic ." He was awarded the 2013 ASA Award for Excellence in Statistical Reporting in recognition of his outstanding work.

(music)

John Bailer: Touchdowns, first downs, blocking, tackles, these are words that everyone familiar with football in the United States will recognize. Chronic Traumatic Encephalopathy (CTE), dementia, memory loss, depression, these are words that recently have been connected to playing football. The emergence of the story connecting football with brain injury is the focus of this episode of Stats and Stories where we explore the statistics behind the stories and the stories behind the statistics. I'm John Bailer. Stats and Stories is a production of Miami University's Departments of Statistics and Media, Journalism and Film and the American Statistical Association. Joining me on the panel today is Richard Campbell, Chair of the Department of Media, Journalism and Film. Today's guest is Alan Schwarz. While working as a journalist for the New York Times, Alan wrote a series of articles on concussions and football and this was credited with helping to revolutionize the treatment of head injuries and exposing the seriousness in them for the athletes. We are delighted to have Alan Schwarz joining us on today's episode of Stats and Stories. Alan, when did you first realize that there might be a story connecting football and brain injury?

Alan Schwarz: Well I got a tip from a friend of a friend. I had been a baseball writer for 16 years exclusively, a very statistics oriented one. I wrote a book on the history of baseball statistics and I wrote a column about sports analytics for the New York Times. But a friend of mine had a friend who had done a lot of research into the concussion area and felt as if there were a serious problem in the National Football League all the way down to youth football that was being, for lack of a better verb, covered up by the National Football League and he asked me to look at some of the evidence that he had and some of the evidence that was emerging at that time… and as a very proud card-carrying math geek, I looked at the numbers and in fact they really did have a problem. It was quite clear that there was a problem to anyone who understood basic statistics. At that time I seem to have been the only one and I ran with it. I brought it to the Times; they let me run a couple of stories and all hell broke loose with the problem of brain injuries and concussions and later life cognitive decline and dementia in football players.

Richard Campbell: This is Richard, Alan. What kind of pushback did you get initially and what kind of challenges did those stories like this pose for you?

Schwarz: Well as I said I was a baseball guy and so I had no particular street cred when it came to football. Now a lot of people think that that benefited me because I had no relationships to lose. I had no press passes to protect. I just wrote what I knew to be true. And did I get pushed? Yeah I guess I did. You know I was so in the trenches that you kind of turn off a lot of about that outside stuff but the National Football League certainly went on a campaign to say that Schwarz doesn't know what he's talking about, what could he know, he's just a math geek. I mean they didn't use those words but they certainly claimed I had no credibility whatsoever and when the rest of the football press didn't follow up on anything, I was really kind of a lone wolf. The fact that I worked for the New York Times gave me all the cred I needed. But I really was the only person who could look at the data, which was incredibly simple to anyone who probably has listened to this broadcast, but among sports writers I was really the only math guy ... I have a math degree from the University of Pennsylvania and I have a library full of books on probability and I knew how to look at the data and it was quite clear that there was a serious problem with dementia and other issues among retired players. And I held on to that belief, more than belief just knowledge, until I proved it to the public's satisfaction.

Bailer: So can you talk a little bit about the data that you had initially, what was the compelling data that led this, that gave you the kind of the real strength of interest in this story and led you to push this forward?

Schwarz: To be honest it was a very basic probability problem, something from Stats 101 in many ways. Basically what we were told was that a disease that is caused only or is known to be caused only by repetitive head trauma, chronic head trauma called chronic traumatic encephalopathy. We were told it's incredibly rare in the general population. I mean at most one out of a thousand, okay. Well the first four men that could only be diagnosed after death and the first four men who died and were examined for it had the disease. Now, it was not a random sample of players, okay, or former players no one could possibly say that. However, four thousand to one shots coming up in a row is pretty darn significant. It makes you question the hypothesis of whether it should be one out of a thousand. It looks like hmm, there's something different about this population. They're not just the general population, something has happened to them which is to say repetitive head trauma.

Well the National Football League and it's hired doctors and frankly a lot of people who were in depth so-called independents who should darned well known better said, "Hey it's just four guys. You can't make any statements about a population as big as former football players" which is about 13-thousand at the time "You can't base statements on only four guys. You don't know the denominator." And I'm like I don't need to know the denominator because we're four out of four for a thousand-to-one shot. And again it's not a random sample but nonetheless no matter how you look at the numbers, it isn't a thousand to one in these guys. It's not obviously a lot higher. There's an increased risk for this disease than in the general population which is a pretty basic concept particularly when you're talking about repetitive head trauma and football. But an awful lot of folks did not find this very compelling. They said Schwarz is just a sports writer, you know. He doesn't know anything. He's not a neurosurgeon. What could he know? It's like well he knows very basic conditional and Bayesian probability and there's a problem here and it's being covered up.

Campbell: In your study of this, how... and it's been awhile since I've read the series; did the National Football League know about this before you started doing this? Had they worried about this or was this something that seemed to be a surprise to them?

Schwarz: Well to be honest there's two different issues. There was the issue of concussions and the long-term effects and dangers of repetitive concussions which are a specific type of head trauma. Now that has been something that has captured the NFL's attention for decades and of course, there's a point of contention over when did they become concerned about it, how concerned should they have been but certainly starting in 1994 when they hired some doctors to look into the issue and conduct so-called studies on the matter. They clearly had a great interest in the possible long-term effects of concussions. After that work was done in 2004-2005 and 2006, the NFL and its committee said that actually concussions have no long-term effects at all so yay for us.

Well around the same time the first two men, retired players were just diagnosed with chronic traumatic encephalopathy which is a very specific degenerative disease that you can diagnose only after death and the NFL became aware of these findings and discredited them as much as they possibly could. The movie Concussion is about that early effort to suppress that early evidence. It was before I showed up. I showed up when the third man was diagnosed. Andre Waters, a defensive back for the Philadelphia Eagles, and that was in January of 2007. And because it was now three out of three, no one had come up negative. All three were positive and even though it's not a random sample we can still question the hypothesis that it's one out of a thousand, then the fourth guy, four out of four and I literally stood toe-to-toe with the Commissioner during an interview and I said, "You know it's four out of four for a very supposedly very rare condition in your players" and he's like I don't... you know he just did not find that compelling, just four guys. What could you possibly know? Again, anyone's listening to this podcast just thinking like duh. But I mean I was the only one who understood the numbers and could read the scientific studies and see the clues that evidence was being covered up, at best misunderstood and at worst covered up.

[music]

Bailer: You're listening to Stats and Stories where we discussed the statistics behind the stories and the stories behind the statistics. The topic today... Concussions in Football. I'm John Bailer, Miami University Statistics Department Chair and joining me is Media,
Journalism and Film Department Chair, Richard Campbell. Our special guest is Alan Schwarz. Now Alan, you've covered a lot of different stories that have a data foundation including baseball reporting, concussions in sports and public health, what are some of the skills that you've needed to make sense of the data that have served as the basis of these stories? You told us a little bit about probability and in essence the calculations that you were doing here we're saying assuming that each of these people had the same risk of this CTE and assuming that they're independent in their responses, how rare is this result? So do you believe that something unusual is happening and their rates are different or do you believe that that something incredibly unusual has happened and that's just life? So you use some probability calculations and thinking but I'm just thinking about what others kind of skills?

Schwarz: Well I think it's a skepticism for numbers and what numbers say and what numbers don't say. In fact, it's my opinion that really taking another step back is that numbers don't mean anything. Numbers means something only after human beings put words around them and so it was when people were saying well it's only four players while only matched with four. If it those are the dance between numbers and words okay...and the tendency for human beings to either misunderstand or misuse those numbers and those words, it's just something I have a very good ear for and so I can tell when someone misuses million and billion instantly. I can tell when... I guess it's hard for me to just explain off the top of my head but I understand how numbers work and the way a lot of people listening to this podcast do. I mean I'm not... Ramanujan. I think though that there's no question that the book that changed my life was Innumeracy by John Allen Paulos which I read while I was in college. He was a Temple University professor. I was at Penn. And it just helped you look at the world really through more of a probabilistic lens, where if something happens and you're told that it's random. Well what are the chances that it was random and what are the chances that there's something different than you're being told? I mean something is as easy as a pair of dice. I mean obviously if you roll a pair of dice... let me give you a perfect example from the election, okay.

I'm on the editorial board of Significance magazine from ASA, and the Royal Statistical Society and you're doing an article on the recent election and how as we all know the pundits up to and including 538 we're saying that there was 70%, 75%, 80% chance that Clinton would win and that there was only obviously a 20%, 25% chance that Trump would win. Well the article we were considering, we were editing, said that the people who predicted it got it wrong. Everyone went bananas, "Oh my god they got it wrong, oh my god how could they have been so irresponsible" and there's a part of me that's saying "Hey if we rolled a die and you asked me is it going to come up one, two, three, or four or five or six which one am I going to bet on?" I'm going to bet on one, two, three, or four. If it comes up five it doesn't mean I got it wrong. It means that I'm just something unlikely happened and so I think we just need to... I want to wrap it up, I'm not doing a great job answering your question but I think it's incredibly important to know that numbers... like I said numbers don't lie. People lie with numbers and when you have enough experience seeing it you can start diagnosing it outside.

Campbell: You know we had your colleague Nick Kristof here a few years ago and he was asking about our journalism program and one of the things that he said is... and I'm paraphrasing him here but when he said at the New York Times we have a lot of very smart reporters, very smart editors but we have a lot of people that don't know very much about numbers and data. And he said if you do anything in a journalism program, you need to be focusing more on data and numbers and helping people write stories that explain those. Now do you find this is still true at New York Times? Talk about some of the things that you think journalists get wrong. There are a lot of... I mean a lot of this is there aren't very many journalists were math majors first of all so it makes you unusual in that way.

Schwarz: Yeah I would say that it's mixed at the Times. Now I left the Times last August... I'm sorry I don't want to misrepresent anything. But I did find that in areas, there was great data sophistication, okay. We have a department that sort of data centric department that would vet things and look at databases and do incredible work, almost forensic work, for example they were the ones who went through all of the Wikileaks cables to find patterns to find things because you had like eight bazillion pages of stuff and did incredible work there. And then you also have The Upshot and the folks who run that. David Leonhardt did it for a long time...very sophisticated folks. I found that that sophistication would generally lie only in those departments. In other departments, there were folks who did not have the instinct for numbers that I did and others do and in fact they had ... I think even some reluctance to trust what ... for example, I knew about the significance of some of the numbers I would come across or the insignificance of numbers and press releases that were just wrong ... and I got a lot of pushback, frankly and it was rather dispiriting. So it's different people ... the different people have views of it. I would say that institutionally the Times is desperately concerned with getting it right, with being very good at everything. I think when you get down to the more granular level of human beings they have different levels of sophistication and trust for numbers that the sports department trusted me implicitly with the concussion stuff. Bill Keller, the executive editor, Jill Abramson, the managing editor said "You know what this guy looks like he knows what he's talking about. He's got a great track record. We're going to back him even if we don't exactly know what he's saying." And it was wonderful but it was a little spotty at times.

Bailer: I'm hearing the sense that a promotion of numeracy is critical in the journalists of tomorrow and I also wonder about the numeracy of the public. You know when you're making the case and telling the story about the rarity of the result, your four cases where each one was ... if the background ... if the base rate was still true of one in a thousand, you're making this argument based on this probabilistic foundation. How hard was it to try to tell that story for a general readership?

Schwarz: I'd say that I did less sort of proving what the numbers meant in my general stories because it wasn't going to work. You know for the layman, it just ... I believe that in the right context I could have done that just fine... but explaining Bayesian probability and blah, blah, blah is not really going to go very far. I think what you do is it's more of a back-end trust among the journalistic operation that, okay look Schwarz is right. There is a heightened risk for football players so how are we going to go about reporting that, proving it, allowing experts to muse on it and maybe even debate it, although how much debate can you have over somebody's contention to two plus two equals five which is what the NFL's approach was, which as we know is not true except for extremely high values of two. But I think that what you do is you put a face on it and when you're trying to illustrate that there's a heightened risk for these problems among retired football players, you put a face on it in the sense that you profile the player, you profile the wife who has to deal with it and you don't do it in a misleading or I guess manipulative sense. You just go into it with the confidence that what you're doing is fair because if there was no heightened risk ... well if you start focusing on the guys with problems it's misrepresenting what the true risk is if they're just the general population. So it was more of a back-end confidence but also I had the ability to read the scientific studies and to diagnose when they were bad and either poorly done, didn't carry the one in a very important spot perhaps intentionally, conclusions were wrong or the good ones. And so the only thing that I would do when it came to the numbers is when for example the number two man, the guy just under Roger Goodell at the NFL, when he said "Hey you know Alan you're focusing on the small number of guys who have had these problems. You know there are thousands and thousands of players who never get dementia at all," which is true. He's not lying. But that's not the point when it comes to epidemiology. I mean there's thousands and thousands people who smoke and don't get lung cancer but you know that's not the point, buddy. And so you have to go out and find an expert who will say that but it's the facility that I had with the numbers allowed me to sort of synthesize all the experts opinions into something that made sense rather than my not picking up on the fact that the NFL was lying to start.

[music]

Bailer: You're listening to Stats and Stories and our discussion today focuses on reporting sports related concussions and other stories that have a strong foundation in data. Our guest is journalist Alan Schwarz, recipient of the ASA Excellence in Statistical Reporting Award in 2013. He produced a multi-year series of articles exposing the impact of sports related concussions. Alan, did you have people trying to argue with you about the interpretation of the data or the analysis? I mean you mentioned kind of the small number of cases that were the foundations of your story but beyond that were there other arguments about analysis that you had done?

Schwarz: Well I'm not sure honestly. There was so much about the basic stuff that I don't even remember their concerns about the more sophisticated stuff. There were chairmen of departments at universities like Johns Hopkins, who would argue with me that I was wrong. It's only four guys who can't make any sense. You don't know the denominator this guy told me over and over and over and I'm like you don't need to know the denominator because the numerator is too high already. And so most people did not see the back-end work that I was doing that gave me the confidence and that sort of was my flashlight in the cave. That's what the numbers were to me. It helped me find the things inside the cave but I didn't talk much about the flashlight. So for example when a study came out showing the rates of dementia among NFL players, a study that was done at the University of Michigan which had been commissioned by of all folks, the National Football League and it found the rates of dementia among retired players to be between five and nineteen times that of the national population. You know a lot of people dismissed it as well it's just a phone survey. But I actually had done the work ahead of time to predict this result and to understand where it came from and why it came out the way it did. So the NFL couldn't brush it aside with me because I already knew the significance of that result and had in fact predicted it almost perfectly in a meeting among editors at the Times to keep me moving on the story. Nobody knows that, nobody needs to know that but again, I think that when it comes to journalism and the numeracy of journalists you really have two different types. You have the types of why are you going to just make a mistake in a story by misunderstanding a number or misstating a concept. That's almost mistakes of the layman. I think what you also want to have is the talents of an expert or at least an enthusiast who can find out good stories to do rather than avoiding annoying mistakes and that takes a long time. It takes a long time and it is a weird mix of being both numerate and literate. A way I sort of describe it is I'm a good mathematician for a writer and a good writer for a mathematician.

Campbell: Very good. Let me ask you about something's changed over the last few months and I think we're dealing with an administration that sort of heightened this notion of denying facts, data deniers; I think this puts a special obligation on journalists who are normally do their reporting and try to take a neutral stance, try to stay out of it, we've used this term objectivity, do you feel like it's time for journalists to fight back? We're seeing a little bit of that just in terms of this overall attack on media, the sort of wholesale fake news accusation.

Schwarz: I don't know exactly what fighting back would entail. I believe you don't get in the street fight with someone with lower standards than you, okay and you don't get into ... it's hard to get in a war of fact versus denial by just presenting more facts because they just keep denying it. The thing that worked for me and I don't know if it would work now but the thing that worked for me is I was incredibly tepid about ... and very just sort of matter of fact and I never was an advocate, I was not fighting back with the work that I did. I was sort of stoically just going hey guys 2+2... I know what you're saying. You're putting all the stuff in scientific papers and you're saying all this stuff. I just want you to know that I know you're lying and I know you're wrong or incorrect and I'm going to prove it. I had incredible confidence because of my math background and it drove them crazy because I never said football shouldn't be played, I never said that these guys should get compensated for their injuries, of course there was the billion dollar lawsuit and all that. I never said that. I just said your assumption that it's the same as the general population is wrong. It's just flat-out wrong and there are great public health implications to that because when the National Football League says hey there's no problem with brain trauma in our sport, it trickles down a course to the lower levels where there is an increased risk as well and different types of risks and different types of assumption of the risk and blah, blah, blah. And so we always treated it not as a professional football issue. Even when we were writing about professional football, it was in the context of public health and whether the public understood the risks that some of them were undergoing and whatever you choose to do with that is your business. I never told a parent. I get asked three times a week ... would you let your son play football or should I let my son play football? I have never once answered that question. And so when you asked about fighting back, I don't know, I would never a million years want to deal with ... for example this administration. My head would explode because you can't fight illogic with logic. It doesn't work and I'm a logic guy so I don't know what they should do. I do think fighting back is pretty hard. I think in some ways where you just have to ride it out. I mean I'm not saying don't do reporting and don't do hardcore reports. In the New York Times they're doing an incredible job and give credit.

Campbell: Just do good work, yes.

Schwarz: Just do as... I don't think we should lower our standards in order to compete with these people. I just don't know if that will work.

Bailer: So one of the things I think about are the consequences of the work that you've done. I mean now that you can reflect back on it, are we now at a point where are all football players having autopsies upon, professional football players having autopsies after death, to see if CTE is present. I mean have there been dramatic changes in terms of the collection of additional data in monitoring this?

Schwarz: Well I think that there is a general acceptance that there are long term risks to playing football professionally because not just the professional game but that means you've played three or four years in college, three to eight or ten years as a youth player and so it's that accumulation of perhaps sub-concussive blows blah, blah, blah. We don't know exactly where these diseases come from. But they're at least as an acknowledgement that you're taking this risk. We're not sure exactly what it is. What the numerical risk is? We probably never will. But at least people know whether they're grown men or parents making decisions for their children that this is out there and it's something you might want to consider, a lot of people going to do it anyway.

Bailer: Has there been data on sort of dose-response that the rate of CTE is lowest among the youth players and then higher among high school only players and then slightly higher among college only players and even higher among professional players? I mean does that data exist?

Schwarz: I don't want to get too deep into the weeds and misrepresented. The Boston University Group would have more full knowledge of that. However there is clearly a proportional relationship between how many years you play and what your risk is, okay. In fact I know that there's some data coming out that the later you start playing football, the less chance you have of having... like getting CTE. But now we're getting down to such granular questions that our denominator really isn't that big. I think it's been 115 NFL players who have been diagnosed with CTE. I don't know what the college and high school only numbers are but they're far less than that. And so the deeper questions we ask, the longer we're going to have to wait but when I was being told, " Alan we need more data to even say there's a risk." I knew that we didn't need more data. The data were right in front of us and I got a chance to put in front of millions of people and work.

(music)

Bailer: That's a great story that you've told Alan. Thank you so very much for being with us today and for the work that you've done on this piece. Stats and Stories is a partnership between Miami University's Departments of Statistics and Media, Journalism and Film and the American Statistical Association. You can follow us on Twitter or Apple podcast. If you like to share your thoughts on our program, send your email to statsandstories@miamioh.edu and be sure to listen for future editions of Stats and Stories where we discussed the statistics behind the stories and the stories behind the statistics.