Episode 8:

Bob Long: It's not unusual for us to hear news reports on medical breakthroughs, especially, for example in the treatment of cancer. Well many of these dramatic stories come from laboratory findings, animal studies, or small studies involving humans, so how do we know which of these new treatments really work? The use of randomized control clinical trials has become very important in fields like cancer research and many others. I'm Bob Long; we welcome you to another edition of Stats and Stories where we look at the stories behind the statistics and the statistics behind the stories, and today we're discussing the issue, "How do I know this medicine works?" Joining me for our discussion, our regular panelists Miami University Statistics Department chair John Bailer, Media, Journalism, and Film chair Richard Campbell. We'll get to our special guest in just a moment, but first we asked Stats and Stories reporter Lucy Borchers to help us understand the complexities of developing new medicines in America.

Lucy Borchers: We often hear about new medicines to help us deal with a variety of diseases and ailments - like cancer, heart disease, HIV, or depression. What most of us don't realize is the length of time it takes to research, test and obtain approvals of the medicines we take. Steve George is a Biostatistics Professor at Duke University School of Medicine. He's been involved in cancer research for more than 2 decades. George has seen great progress in discovering safe and effective therapies through clinical trials, which can take ten years to complete. Clinical trials are the best tool for obtaining reliable data on whether a drug is safe and will produce the results it's designed to achieve. Historically, George says clinical trials have been used to determine if one medication is better than another for broad groups of people battling a certain disease. More recently, George has seen a shift toward targeting therapies in a more personalized way.

Steve George: Not all patients respond alike, and the question is why is that? Can we identify factors that would be predictive of success for a particular therapy - know that ahead of time -we can target those patients for specific therapies. It's an appealing concept. It's still elusive in practice because of the complexity of the disease and all of the many factors that go into that.

Borchers: Companies seeking approval of new medicines must gain approval from the Food and Drug Administration. Lisa LaVange is the director of Biostatistics in the FDA's center for Drug Evaluation and Research. She says experiments with new drugs start with animal testing...then move into an initial clinical trial for humans. That's followed by studies to determine the optimum dosage and the population best suited for medications. LaVange says in the final stage before approval companies must do two clinical trials that are randomized and well-controlled.

Lisa LaVange: They have a number of aspects that make them well controlled. If possible, the trials are what we call blinded or masked so that neither the patient nor the physician enrolling and treating the patient or anyone involved in the study knows which treatment the patient is on.

Borchers: Lisa LaVange says her counterparts in the FDA's Medical Review Office look consider safety issues for a new drug. Before approving a new drug she says the FDA must assess benefits versus risks.

LaVange: So what we want to know at the time of approval is that the benefits outweigh any risks involved with the drug. It's hard to say there's no risk whatsoever with the drug. There are always risks that are with drugs. So we have to make an assessment that the benefit the drug affords to the patient outweighs the risk. And then if there's a serious risk with the drug, it may result in non-approval.

Borchers: Duke professor Steve George says one important step in assessing risk is having an independent data monitoring committee get involved since it has no stake in the outcome.

George: It is certainly true that these therapies for any disease can have adverse effects and sometimes fairly serious adverse effects. It's a difficult matter to weight that relative to the potential benefit of the therapy.

Borchers: Both Steve George and Lisa LaVange say clinical trials are the cornerstone of new drug development. LaVange says FDA also assists in developing labels that help medical professionals explain to patients what effects the new drug will have. For Stats and Stories, I'm Lucy Borchers.

Long: Our special guest today and we welcome Marie Davidian, William Neal Reynolds professor from North Carolina State University, also a past president of the American Statistical Association and her expertise involves applying statistical methods to challenges in the whole field of health science research. I mentioned clinical trials which are a big part of what we hear about today and I just wanted to have you talk about your involvement in this field from the statistical point of view.

Marie Davidian: Okay, thanks Bob. Thanks for inviting me. The controlled clinical trial is probably the, I think you could call it the bedrock of the evaluation of new treatments. It's the gold standard by which the Food and Drug Administration, which is the regulatory body in our country, evaluates and approves products for distribution in the marketplace. I've been involved in health sciences research in a variety of disease areas: HIV infection, cardiovascular disease research, cancer for over 20 years and I've participated in the design and analysis of these studies through my involvement and collaboration with those scientists.

Long: John Bailer, we go to you for the next question.

John Bailer: There's a lot of vocabulary that comes up and you talk about design and analysis and trial and there's a technical meaning that maybe is confusing when people hear it, so why is it called a trial and when you say its design, what are some of the aspects of design that you have helped with?

Davidian: Okay, I think it's called a trial because in some cases, the first time a treatment is evaluated in the sense that it's being compared against maybe the current standard of care or against nothing in the case where there isn't a current standard, so what would be called a placebo. So a clinical trial is actually a very precise experiment to gain appropriate evidence to determine whether or not evidence exists. In fact to say that there is a real difference.

Long: We've been doing clinical trials in medicine for decades, but I just wanted to get your feel on how rapidly things are changing today because it just seems like things have really exploded in this whole field.

Davidian: Indeed, the controlled clinical trial, I think since the early 1960s when legislation in Congress was enacted to require substantial evidence from well controlled studies. The Food and Drug Administration has required that such studies be conducted by what are called sponsors, pharmaceutical companies and these days biotech companies. What we're seeing now more recently, over the past decade in particular with the advent of the Human Genome Project, is the desire to modify and expand trials to say, incorporate genomic information for example, on individual patients that could be used to in fact guide their treatment. So rather than just looking very broadly at "is treatment A better than treatment B?" in a very broad sense, trying to develop new sorts of strategies for conducting these studies to bring the genomic information in and try to target the treatment to the patient.

Long: Richard Campbell.

Richard Campbell: I know that some of your work involves personalized medicine and I'd like you to talk about the sort of gray area between coming up with data that's meaningful and also the sort of subjective part of making decisions about what your medical requirements might be.

Davidian: Okay, well in a clinical trial, we try to be as objective as possible, so in fact, the basic set-up is patients are randomly assigned to receive one treatment or the other so that there will be no unconscious or conscious bias for example, that one treatment gets assigned preferentially to say, the sicker patients, which would make it for an unfair comparison. So in the realm of personalized medicine, what we would like to do, ideally, is to develop sort of objective evidence-based say, rules, if you will for dictating which treatments ought to be given to which patients based on their personal characteristics. Now of course, the way clinicians practice medicine, they make decisions like this all the time; they take, as input if you will, the information on a patient, synthesize it, and make a treatment decision among the available options, which one should be given to this patient? Now obviously that involves a certain amount of subjectivity as well, so I think the challenge is clinicians need to use their clinical judgment and their experience, at the same time there's a desire for ways of giving people treatment that are objectively based on evidence. And so the challenge is how do you integrate those two perspectives and that's where we sort of stand right now.

Bailer: You know, some people might wonder, why do random assignments? What's the value of doing a random assignment? Maybe if you have knowledge, shouldn't that make it better? What's accomplished by doing this?

Davidian: Okay, well we're in the business here in a clinical trial of trying to learn which of, say two treatments, is better, better in what sense, better perhaps on average if it were given to the entire population of patients. Now if patients and clinicians were to choose their own treatments for example, a clinician might have a preconceived notion that perhaps the new treatment is more likely to benefit patients who are sicker, and so might tend to give that treatment to those sicker patients. You can think about that that would allow an unfair comparison because sicker patients are being assigned the new treatment in greater numbers and that could very possibly make that new treatment look worse than it actually would be if it were to be given to all the patients in the population without regard to their health status.

Long: You're listening to Stats and Stories where we're looking at the question, "how do I know this medicine works?" I am Bob Long; our two Stats and Stories panelists are Miami University Statistics department chair John Bailer and Media, Journalism, and Film chair Richard Campbell, and our special guest today, Marie Davidian, the past president of the American Statistical Association; she's also the William Neal Reynolds professor of statistics at North Carolina State University and her expertise involves the challenges we face in the whole area of health science research. We also wanted to find out what people out there know about our topic, so our Stats and Stories reporters asked them the question, how long do you think it takes to develop a new drug or a new form of medical treatment today?

Woman on the street #1: I would have to say seven to ten years, and that's probably because they have to go through not only proposing it, but then actually going through the study and finding the answers and then they have to actually get approved by different federal companies.

Woman on the street #2: I would say you'd have to have years of studying and testing and approval.

Woman on the street #3: I would have to say 10 plus years because you have to go through everything from beginning to the end and really do the case studies and make sure the case studies are on larger amounts of people versus a smaller group so you know that it has been tested on a wide range of people with diseases or whatever they're looking for.

Woman on the street #4: My guess would be that it would probably take around 10 years.

Man on the street #1: Twenty years.

Woman on the street #5: Twenty years.

Long: Marie, I guess that's a great question to ask you based on the answers we've heard. How long do you thin, or how long does it actually take to develop a new drug or treatment today?

Davidian: Okay, well the answers you see out there vary, but I think it's pretty much accepted that from conception, at the very earliest stages in the laboratory through the entire process of testing and evaluation to approval for the marketplace by the Food and Drug Administration, on average takes perhaps 10-12 years.

Bailer: And how many drugs would make it through? How many start at that very first part and make it to the end?

Davidian: I think the statistic is something on the order of 5,000 that enter into early testing, maybe 1 is eventually approved.

Bailer: So what are some of the phases of this approval process? It's certainly a staged exercise; you're probably demonstrating something has the potential to be useful, so what are some of the studies that are conducted along this path?

Davidian: Okay, well let me just start off with the very early studies. This is before a potential treatment is even given to say animals or eventually humans. The early stages involve laboratory experiments for example, going through vast numbers of chemical compounds, in the case of drugs, looking for particular compounds that might exhibit certain kinds of biological activity that might be thought to be associated with various aspects of the physiology of the disease under study. So those studies will take place very early and do involve statisticians that help to design them. After that, when a chemical entity is identified as potentially promising, then there will be additional laboratory studies and eventually animal studies to look at things like, can this chemical compound actually be introduced and absorbed into the system? Forget if it does anything, can we even get it in there and then furthermore, is it safe? Are there adverse effects that possibly result from administering it? So those animal studies take place very early before even the FDA is really involved heavily.

Long: Richard Campbell, we'll go to you for the next question.

Campbell: Marie, I wanted to ask you about the sort of, back end of all of this, when you finish the study and you're in the position where this gets transmitted to the general public and it's off in the job of the journalists to do that. Could you talk a little bit about what journalists do well and not so well when they're interpreting the work of statisticians?

Davidian: That's a loaded question. I've actually been very impressed, particularly in the past several years, with reporting of statistical information in the media by journalists. It's difficult to understand much of it because, for example, a natural thing for a member of the public to ask is, how can you look at a drug in maybe a thousand people and actually conclude that that drug is better than say, what's currently available, and that requires understanding of basic statistical principles such as taking a sample from a population, the sample being sufficiently large, that you have enough confidence in the resulting evidence from the study to conclude, you can never be totally certain, that's what we statisticians are all about, but that you actually can conclude with a high degree of confidence that the result is something real and not just something spurious.

Campbell: I was going to ask, too. Specific things that I notice when I work with our journalism students are things like what we sometimes call false-balance where - in journalism you're asked to go out and tell two sides of a story, well sometimes there aren't two sides and sometimes I notice journalists sometimes have trouble distinguishing correlation from causality, so maybe you could speak to those issues. If I could talk a little bit about the false balance thing has to do with like climate change, we've talked about that on this program where if you get somebody that has done research on climate change, you're supposed to go out and get an equal number of folks who will dismiss it and that's actually an example of false-balance.

Davidian: I see, I think in this realm the issue of correlation versus causation that you brought up is a key. Maybe, I don't know false-balance is quite what I'd call it, but it's a bit controversial in terms of understanding what evidence is out there. So for example, we harken back maybe 10 years, hormone replacement therapy had been highly touted for women, post-menopausal women, to be very beneficial and that conclusion was based on many studies, which statisticians call observational studies. These are studies where all you really do is observe which treatments were given to which patients over time, so for example, some women got hormone therapy, some didn't and as a result of those observational data, it was concluded that hormone replacement therapy was beneficial. Later on, a clinical trial, randomized clinical trial was conducted, the Women's Health Initiative, and it turned out - so women were randomized to receive hormone replacement therapy or to receive a placebo, so an inactive substance that looked like hormone replacement therapy, and as it turned out, the study was stopped early because there were so many cases of adverse effects like breast cancer and so on. Well this is correlation versus causation in action. As I said earlier, a randomized study allows for an unbiased evaluation of the two things being tested. So another way of looking at that is that it allows you to feel comfortable in inferring causation. The observational studies on the other hand, where patients and clinicians were choosing their own treatments, that could have been done preferentially, and while we do have statistical methods to sort of disentangle that, you can never be certain that you've actually done that. So as a result, the correlation, say between hormone replacement therapy and beneficial health outcomes that was observed can never really for certain be attributed to the hormone replacement therapy.

Bailer: I'd like to revisit a word that you used a couple of times, that's that design thing. I think randomization is one aspect of a design. Other things that we've talked about in other contexts are replication, control, and even the generalized ability of the population under study for a target population of interest. If you involved in helping someone design an experiment to evaluate a new drug or new treatment, what are some of the decisions that you would have to help them to make?

Davidian: As a statistician?

Bailer: As a statistician.

Davidian: Okay, well, and I will just add that as a statistician involved at the very earliest stage of the design is very important because we can contribute in the ways I'll discuss. So first of all, we have to understand the subject matter and discuss it with our subject matter collaborators to even try to understand what is the comparison of interest that makes the most sense? So if there's a new treatment, to what should it even be compared? That's more of a subject realm issue, but it's one we have to understand, we all have to agree on. But once we've established that, then the issue becomes, okay, what would be a meaningful difference to a clinician in terms of, if I can make the average outcome, maybe the average survival time increase by X number of months that would really be an important breakthrough. So I have to understand by talking to my subject matter colleagues what would be an important difference to even detect. Then, we have to decide, well how many patients would we need to have to involve in the study to gather enough evidence to feel confident that we were able to detect such a difference, if it in fact exists. So these are all considerations we have to take into account when we design one of these studies.

Long: This is Stats and Stories; I'm Bob Long. Our question today is, "how do I know the medicine works?" Miami University Statistics department chair John Bailer and Media, Journalism, and Film department chair Richard Campbell are here asking questions of our special guest, Marie Davidian. She's the William Neal Reynolds professor of statistics at North Caroline State University and her expertise involves challenges in health science research. We wanted to also know how much people know about today's topic, so our Stats and Stories reporters asked them, how much money do you think it costs to bring a new drug to the market today?

Woman on the street #6: Three million dollars. I've seen different things between especially if it's about cancer, it's probably going to be more than that. But in the millions or billions I've seen numbers like that, but I've never actually done any studying on the topic...so.

Woman on the street #7: I would say millions of dollars just because if you think about the time that the researchers are putting in to it, all the people that they're paying to develop those and do the testing. I mean it would definitely add up over the years.

Woman on the street #8: I'm going to stay with the tens and say 10-million plus to do all of the testing, to look at the drugs, test more side effects, see if there's anything that comes with that.

Woman on the street #9: I would assume like millions to create it and like bring it to market.

Woman on the street #10: I know that it probably cost a million dollars to develop one of the drugs that I'm on because I know it's expensive right now. And I know part of that is recouping the cost of the development of the drug.

Long: Marie, I did want to ask you that question because I think people would be really curious just how much money is involved in this realm.

Davidian: Again, I believe estimates vary, but I think for a pharmaceutical product, so a chemical entity produced by a pharmaceutical company, the estimates range from maybe $800 million to $1 billion.

Long: And that brings up another question I have because I used to teach a course in writing for advertising and one of the things my students were always curious about was why do we do so much pharmaceutical advertising, I think the United States, as I remember is one of only two nations, I think New Zealand is the other, that allow drug advertisements and so I think sometimes when people see all these new medications and then they hear at the end of the ad what the side effects are, they sometimes wonder, I could die from this or it could cause cancer, which is what I'm trying to prevent, so it's kind of curious when we hear all this. I'm just kind of curious about, is that part of the reason why we do this, because of the cost to bring this to market so they also have to get the word out there to people to try it once they actually bring it, once the FDA approves it.

Davidian: Certainly I think that's part of the motivation; of recouping those research and development costs is obviously something a company wants to do, needs to do. I think there's some debate about how we go about doing this and I don't think we want to get into that here, but indeed; it's a very expensive process conducting a major clinical study, a major clinical trial costs millions and millions, tens and hundreds of millions of dollars. These are very, very complicated studies.

Long: Yeah, John.

Bailer: I'm curious; given what you know about clinical trials and the work that you've done, how has this changed your behavior as a patient?

Davidian: That's a great question. I guess, you know, when I visit my clinician for my yearly physical or maybe not quite yearly, sorry. You know, I tend to ask him questions that sort of come out of my experience in dealing with clinicians and my dealing data and so on. I ask him about recent things in the medical literature that might be relevant to some ailment that I'm experiencing or whatever. I think it's made me a little more, I guess aware of what resources and evidence might be out there as a result of these studies and gives me an inspiration to ask more questions of my clinician.

Long: Richard.

Campbell: I'd like to shift gear here a little bit and talk, John's brought a number of guests on the program, women statisticians, in journalism, in terms of our majors here for instance, we have a lot of women students, probably 75% to 25%, and I would guess that this has changed in your time as an academic statistician. Could you talk a little bit about opportunities for women statisticians and how much change you've seen over time and what's facilitated that change?

Davidian: Certainly, when I went to graduate school in statistics at the University of North Carolina at Chapel Hill in 1982, there was one female faculty member, she had just joined the department and she stayed for maybe five years then left. I was one of two female graduate students, so we were definitely in the minority. Over the years, I think as with many disciplines, more opportunities for women have come about. One particular area of statistics where I've seen a huge amount of women entering the field is in the health sciences, what we call biostatistics. There are departments of statistics and departments of biostatistics at universities and departments of biostatistics focus specifically on training for entering health sciences research and applications. Maybe it's because, I don't know, maybe women gravitate more toward that, but we've seen an enormous increase in the number of females entering those graduate programs. Today for example, in my department at North Carolina State, we're a very large department so we have both a biostatistics group as well as statisticians engaged in other areas of application. About 55% of our graduate population, graduate student population, is female and at least a third of our faculty, in fact, my department head is female. So I've just seen an increase; in more theory oriented departments, I've seen less of that. In the faculty ranks, at the more senior levels, women are not as highly represented, I think mainly because of my experience as a graduate student at that time, but at the lower ranks, the assistant and associate professors, women are definitely very well represented on our faculty. So I think we're one field where there are lots of opportunities for women.

Long: John Bailer, we've got time for one final question today for Marie.

Bailer: Well to follow-up on sort of the future, moving into careers in this, if someone was interested in doing what you do, what kinds of things should they be studying as an undergraduate?

Davidian: Statistics is an interesting field. There's a famous statistician named John Tukey, who has long since passed away, but who was at Princeton who once said, "The great thing about being a statistician is that you get to play in everybody else's backyard." So the exciting thing about statistics is we get to learn a lot about other sciences, and we need to do that in order to be effective collaborators, but at the same time statistics is grounded in mathematics. We are not mathematicians, per se, but we use mathematics heavily in our work. So a student thinking about going to graduate school in statistics or entering the field even as an undergraduate would want to get a very good training in basic mathematics: calculus, a course called linear algebra, certainly probability; for entering graduate school in a PhD or Master's program, that would be the minimum requirements. At the same time, learning something about various sciences for example, biology, genetics if you're interested in health sciences research would certainly be important as well. So getting a sort of well-rounded training, but definitely have the mathematics background.

Long: Marie Davidian, we want to thank you very much for sharing your expertise with us today on Stats and Stories.

Davidian: Thank you.

Bailer: Thanks, Marie.

Long: If you'd like to share your thoughts with us on our program, you can send your email to StatsandStories@MiamiOH.edu. Be sure to listen for future editions of Stats and Stories where we'll talk about the statistics behind the stories and the stories behind the statistics.