Okay, welcome back. This is a supplemental lecture for unit 6 and I wanted to kind of gather some of my thoughts on what I'll call, or what is actually called the replication crisis now in science, and how it applies to research that actually we've discussed thus far in this course. And I said I'd come back to this, and the question is well, we've already talked a little bit about this. Why did the standards et al and Chabris et al., large scale candidate gene studies, why didn't they work? Everything in the literature up to the time they did those studies would say they should have been successful. But they found nothing, they were exquisitely well-designed studies, based on what was thought to be a very strong hypothesis emerging from the literature, that yielded absolutely nothing. There's something that needs to be explained there, I think. Now, I showed you this slide before there are various explanations that people have offered for why they didn't find anything. I want to address one of these right here, at the beginning. One explanation is, well, maybe there's nothing to actually be found. Maybe schizophrenia and general cognitive ability are not heritable. And in fact some of the critics of the behavioral genetic or the twin and adoption literature have actually said that. They've said that, well, you know, the failure to find the underlying genetic variance may be telling us that, that the twin and adoption studies are wrong. I actually don't believe that this is a viable explanation for many reasons, one reason is, in my opinion, and hopefully I've convinced some of you, that, that, the twin and adoption data are just too massive, too coherent to just completely dismiss as meaningless. And secondly, is this is actually a quote from a critic of the literature, and one of the things he was criticizing in 2013 is the failure to find genetic variance for schizophrenia. Well the problem with that is that in 2014 in the study we talked about last week, the large scale GWAS actually found over a hundred variants, genetic variants, sequence variants that were associated with schizophrenia, so there probably is something inheritable there. We haven't found much for general cognitive ability, admittedly but I'm confident as the sample sizes grow and these massive GWAS's, we will find something. So if it's not, if, if I don't buy that explanation, what do I think are the major reasons why those two studies didn't find anything? And I've already given you what I think are the major reasons. One is that their sample was not large enough in both of these studies. The, the, the Chabris study was 10,000, that seems pretty large, right? And the, the Sanders et al study involved 2,000 people with schizophrenia, 2,000 controls by the standards of those of when they did those studies those seem to be extraordinarily large samples. But right, one of the things that we're learning from GWAS is that the genetic variants that we're trying to identify are very small. So those samples are probably not large enough to identify things, even if they're there and that still doesn't explain how it is that the literature that they are reading said these are the genes to look for and yet they couldn't find anything. Why were these not the right candidate genes? And therein I think comes the problem of what's called the replication bias. So first of all, a little terminology, when you're doing a statistical test, sometimes people doing statistical tests differentiate a true positive result from a false-positive result. A true positive result is when you say something is statistically significant and in fact there is a real effect there. When we say something is statistically significant we hope it's a true positive result. We say that it is a significant result, and actually that is an, an allele that's associated with the phenotype. The alternative possibility, though, and it is a possibility, is that when we say something is statistically significant, there's actually no effect there. It's a false positive. There's always some chance that that would happen. We try to minimize that chance, but there's always a chance that there's a false-positive in the research we do, but we, right? If we're do, if the researchers are doing good science then we try to minimize the chance that when we say something's there that's it's false, that it's a false-positive. Unfortunately, at least in the first generation of candidate-gene research, research, researchers and scholars have come to the conclusion that most things that were called significant were probably false positive results. This is a review article by Colhoun, et al pblished in Lancet in 2003 they actually based on their analysis, estimate that 95% of things that were called significant were false-positive results. So if they're false-positive results you don't expect anybody to replicate it. There's no, if, if Saunders is basing the design on their study on things that are false-positives there's no reason to expect that they would find anything. Here's an illustration of what geneticists now think was going on with these candidate gene studies. What this is a, is from a paper by a, a very prominent researcher that I'm going to talk a little bit about here in the remainder of this supplem, supplementary lecture. John Ioannidis and I put a link, I've actually put a link to the article, to an article by him on the course web page that you could read, it's supplemental. He, he's written a lot about this, he's well worth reading, and what he's done here, this is based on candidate gene research and I think there's about seven or eight different candidate gene associations he's talking about here. Some of them are behavioral, so you can see one involves schizophrenia, but some of them are non-behavioral, lung cancer. And what he's plotting here is the effect for the genetic association as a function of the sample size in a meta-analysis. The effect here is on an odds ratio schedule scale. So if you recall the odds ratio, if it's one, there's no association. If it's greater than one, then carrying that genetic variant or allele increases your risk of of, of having the disorder, whatever the disorder is. If it's less than one, then carrying the allele is actually protective, it reduces your risk for the disorder. And what he's doing here is just take this one here, this green one. He plots what the original odds ratio is in the first published study, and then he combines that study with the second study and now the sample's up to 200, I guess. And he recomputes the average odds ratio and then he does it when he adds the sample for the third and the fourth and the fifth and the sixth, and he's doing this for all the different genetic associations. Well, if you look at this, what's the overall trend? As you get more data, the evidence for there being a genetic association becomes weaker and weaker. If there were a real association there, wouldn't you expect is, as you added more data the evidence would get stronger but, in fact everything is regressing back to the null value of one here. If it looked like a risk real before, it becomes weaker and weaker and closer to one. Over time if it looked like it protective it becomes less protective as you add in more data. It looked like across these all these different studies people were looking at false-positive results. This is what Ioannidis found when he looked at genetic association studies. This is the paper that that I'm linking actually on the course website. In fact he's claimed, and it's well worth reading the paper if you're interested in science, he's made the claim that the problem of false positive findings is actually not specific to psychology or genetics, that it's a very broad problem within the sciences today. That he claims that most research findings that are published are false and they're false for various reasons but he's identified six factors that he believes actually promote the publication of false research findings. And Ioannidis has been so influential on the practice of science over the last five years that research journals have completely changed their editorial process to try to iden, to try to address the problems that he's identified. The things that lend themselves to the publication of false positive findings are having small samples, having small effect sizes. Having a large number of potential associations you could look at. Having flexibility in the design of the study. How do you exactly measure that phenotype? Do you transform the scale? Do you eliminate that observation because you think it's an outlying observation? Those are all flexible decisions that researchers make. Many research teams competing, it's a popular area of research. And finally the topic is, has potential political or financial benefit. If you look at candidate gene studies, I've checked off the ones here on Ioannidis list that actually characterized that first generation of candidate gene studies. The samples were typically small. We know the effect size they were pursuing were, were small. There were a large number of potential relations they could be looking at. They had flexibility in the design of the studies and there were many different people that we're competing to actually publish because of, it broad reputation, it enhances your reputation in the field. [SOUND] Well, how is it that science actually gets misled? There's I think a beautiful illustration of this from one of my favorite scientists. So we're getting really afield from genetics and psychology now. But many of you probably know Richard Feynman. Richard Feynman is a Nobel Laureate, he's a physicist, he's passed away. He's a physicist, he won the Nobel Prize I think in the 1960s at some point. Very, very famous scientist and he gave a famous speech called Cargo Cult Science. And I'm not actually going to define Cargo Cult Science for you here, I, I actually linked the article on the course website. It's actually an interesting article. It's a very short article, it's about three pages. It's actually a commencement address he gave at Cal Tech University in 1970 something. But in this article he talks about another Nobel laureate in physics a man you, you may know I, I'm forgetting his first name I think it's Robert Robert Mikin is also an American who won the Nobel Prize in physics in the early 20th century he would have won the prize. I believe he won the Nobel Prize, I'm pretty sure of this because he figured out that electrons had a negative charge. Milken did a very famous experiment called the oil drop experiment and actually a lot of school children repeat the experiment. And when Milken did his experiment, he actually, when he published his results, he had made a mistake in the write up of his results and, and Feynman in this article talks about Milken's mistake. And it was kind of a in, in some sense a minor mistake. He did the experiment right, but in writing up and in analyzing his data, when he was deriving the equations, he inserted the wrong value for a parameter in the equation. So, Milken then because he derived the, his results using this wrong parameter, he didn't get the right result, but it got published and it got in the literature. And what Feynman asked about in his talk here is, okay Milken, this very famous genetis physicist publishes this wrong result but what happens? What happens to that wrong result when it gets in the literature? We know that science is self-correcting, right? But this is clearly a false positive result. It's wrong, you would think that the next person coming along would, would correct the mistake and get the right answer in publishing it. But that isn't what happened. If Milken got this answer over here, and this is the right answer, the next person who came along didn't get the answer here, couldn't get Milken's answer, right? Because that's the wrong answer. But somehow they got an answer kind of close to Milken and then the next one got an answer maybe a little bit closer to the right answer and then again and again. And so over time, they got the right answer, but it took multiple attempts to do that. So what must have been happening? But we can only conjecture because we weren't in those labs. But it's not hard to conjecture that what must have been happening is that a professor told the graduate student, or the postdoc, or the lab tech, go out and do Milken's experiment and bring me the results. And so when the first person went out to do that, to try to replicate Milken's results, they brought back the wrong result. And the professor looked and said, that's not right. That isn't what Milken got. We can't publish that, Milken's got a Nobel prize. How can we publish that? You gotta go back and redo things and get the right answer. So they go back, of course, they never can get the right answer. But in those flexible designs they change things and they at least get something that's comfortably close to Milken. What happens in research, and, and some people complain that I'm kind of nihilistic. I'm actually, I actually am not, I think that science is the best way of knowing things. But I also think science is a human endeavor, and as in any human endeavor, it's a flawed endeavor, and we need to understand those flaws to correct them. And it's those flaws that led us astray in the candidate gene approach. What was happening in the Milken example, and probably happens in my career, I'm sure I'm as guilty as anyone else over the years, but I try to be more vigilant now, is we expect certain things when we do a study. If we don't get what we expect, we go back and we try to re-analyze our data to get the right answer before we publish it. Maybe the right answer is just getting something that's statistically significant because the journal won't publish it otherwise. But we might reanalyze our data every which way and we don't write it up that way, so people reading it don't understand. What we've learned over the last ten years in genetics, and in other areas of science, is that it is sometimes called the self-confirmation bias, that we all hold. And that if we want to be good scientists, we have to combat that self-confirmation bias. John Ioannidis, Richard Feynman have done a lot in revealing to us our biases. Now it's up to us to try to address those. Today I think people are making progress in genetics because they've looked at themselves critically at what they were doing at, at why the candidate gene approach failed and tried to address that in the new round of GWAS research. So if I go back to that list that Ioannidis has identified to try to id- and, and what we now or what now geneticists are now doing in GWAS to try to limit the extent to which these factors might bias us in doing our research. They really have, I've underlined the ones that GWAS now, I think, have minimized. The samples aren't small anymore, in fact, they're massive. 100,000 people, 200,000 people. The effect sizes are still small, we're, nothing we're going to do about that. Now we actually say before we do the study, what, how we're going to analyze the data. So we're not analyzing the data every which way. So we pre-specify the analysis to try to constrain us from trying to engage in this self-confirmation bias. That also eliminates the flexibility in the design. Everything is, kind of, worked out ahead of time. If you want to say something significant in a GWAS, you better get something that's signficant at five times ten to the minus eight. Finally, and this is actually, maybe, fortuitous, even though it's a popular area of research there aren't multiple research teams competing anymore. Why aren't they competing anymore? Because you can't compete with a sample size of 2,000. You need a sample size of 100,000. You're not going to com, amass a sample size of 100,000. You're going to have to cooperate with that scientist you used to compete with. So people are actually, and I think it's an extraordinarily wonderful development in the field. People are now pulling their resources, reducing the competition, eliminating this source of bias. So I think we're making progress, though progress might be glacial, it might be very slow, but it's, I think, real progress now. Thank you very much.
