Now I approached you and say, hey, I've got a fun way for us to pass the time. Here's a fair coin. I'll toss it and let's bet on the outcome. Just for fun. I'll take heads and maybe we can play for $10 per toss. Just to make it interesting. You toss the coin, it comes up heads. You toss it again, heads, third time heads again, fourth time heads. How long before you begin to wonder whether the coin really is fair. How long before you're convinced that it's fair or not fair? At some point, you might start doubting that the coin is fair. But at what point can you be certain? The simple fact is, you can never be 100 percent certain that this coin isn't fair. But you can determine how unlikely it is, that you would get the observed outcome. In this case, four heads in a row. If the coin really were fair. When this is really unlikely, then you can be confident. But again, never a 100 percent certain, that the coin isn't fair. Let's think about this in more detail. Here's a chart showing for a fair coin, the likelihood of having different outcomes after a number of flips. Notice, that it looks suspiciously like the probability chart from the last video. In fact, it is the probability tree from the last video. Remember, that told us, what we should expect if we knew that the coin was fair. Now, we don't know whether the coin is fair, but you can compare the outcomes from our coin flips to this chart, to see how likely it is that you would see these outcomes, if the coin were indeed fair. If the coin were fair, then they're still at 25 percent chance that you would have zero tails after two coin tosses. There's a twelve and half percent chance you would have zero tails after three coin tosses. There's a 6.25 percent chance, that you would have zero tails after four coin tosses. There's a 3.125 percent chance, that you would have zero tails after five coin tosses. Just over one and a half percent chance, that you'd have zero tails after six coin tosses. It should be clear by now, that this chance will never get all the way to zero, which is why you can never be completely certain, that the coin isn't fair, even if you get a 100 tails in a row. But at some point, the preponderance of the evidence should make you pretty sure, that the coin isn't fair. Specifically, at some point we say, if this really were a fair coin, then the probability of getting this many tales and our coin flips would be ridiculously small. Therefore, I'm confident that this is not really a fair coin at all. In statistical terminology, we start with a hypothesis that the coin is fair, and then we test that hypothesis, by generating data, the coin flips and assessing how likely is it, that we would observe these data, if the hypothesis were true. If it's very unlikely that we would observe these data, if the hypothesis were true, then we reject the hypothesis. This example, there's a twelve and half percent chance of getting heads on all of our first three coin flips. So if I get three consecutive heads, then I'm twelve and a half percent confident, that this coin is fair and 87.5 percent confident that it's not fair. If I get six consecutive heads than I'm 1.625 percent confident that this coin is fair and 98.375 percent confident, that it's not fair. In many statistical analyses, we use five percent as the threshold for rejecting hypothesis. So if we flip the coin six times and got zero tails, then we would reject the hypothesis that the coin is fair because there's only a one and a half percent probability, that we would get that outcome if the coin we're really fair, and this is less than five percent. Put differently, we would be 98.375 percent confident that the coin is not fair and that's higher than 95 percent. But if we flip the coin six times and got one tail, then we cannot reject the hypothesis. Because there's over 10 percent chance of getting either one-tail or more extreme outcome that is zero tails. Even if the coin we're really fair, you may be surprised that we use the probability of getting one head or more extreme, rather than simply the probability of getting one head. The handout explains why, just in case you're interested in this. But for our purposes, we only need to have the basic intuition behind statistical inference and hypothesis testing. One final comment. So far, we've only considered relatively small samples, you know up to six coin flips. However, for reasons discussed in the previous video, in quantitative analysis, it's important to work with larger samples if possible. Remember how we saw that lop-sided events are less and less likely the more choices we make. That the tosses are more and more likely to converge on the true probability of tossing a tails. This means that for example, if you flip a coin 25 times and see 20 heads and five tails, you can be more confident that it is unfair. You flip it only five times and see four heads and one tail. In both cases, the coin comes up heads 80 percent of the time. But when that happens over a greater number of tosses, it's telling us more about the true nature of the coin. Even more valuable though, is that with more tosses, we can be confident that the coin is unfair, with a much more modest deviation from 50 percent tails. Recall that if we flip the coin six times, then with zero tails, we can be confident that it's unfair, but with even just one-tail, we cannot. If we flipped that same coin 30 times and get five tails, then we would be 99.9 percent confident that it's unfair. In fact, we could see as many as 10 tails out of those 30 tosses and still be 95 percent confident that the coin is not fair. If we flipped it a 100 times, we can see 40 tails and still be confident that the coin is not fair. That's why it's beneficial to collect a large amount of data, when you do quantitative analysis, and it should fit with your intuition as well. Imagine that there are three job searches in your organization, and in each case, the search committee selects someone with certain characteristics. Given the topic of this program, let's say that each time the job recipient is a man, you might wonder if a coincidence, but at this point it's difficult to identify, just from numbers that this is more than just a coincidence. Now imagine that the organization fills 50 slots and 40 recipients are men. Intuitively, you probably sense that this conveys a stronger pattern than mere coincidence. Quantitative analysis allows you to codify that intuition in a useful way. We will come back to this idea later in our time together, when we discuss how to interpret our analysis.
