Introduction to experimental design

- [Instructor] What we are going to do in this video is talk a little bit about experiments in science and experiments are really the heart of all scientific progress. If you think about, let's just say this represents just baseline knowledge and then people have hunches in the world and for a lot of times people say, hey, I have a hunch that that thing is good for you or that thing is good for you but they really had no way of measuring how confident they were. They really had no good way of proving it and even more, because they had no good way of proving it, it was hard for people to build on top of that knowledge but with the scientific method and experiments, people were able to say, hey, we have a hypothesis here and we were able to do some well-designed experiments and so we feel pretty good that this is true and then future people are going to say, hey, since we feel pretty good that this is true, maybe we can design experiment to see whether that is true. Hey, that actually is true and then they can build on that and we end up having scientific progress that can accumulate over hundreds of years and this is really important that the experiments are well-designed because in the future and this happens all the time, we might realize that hey, actually there was a little, a few assumptions baked in here that weren't accurate that allowed us to make essentially misleading conclusions. So our conclusion wasn't quite right there and then we will have to rebuild from that point in order to make sure that we are truly making progress. So the key question is how do we set up well-designed experiments and it's a whole field of study but the whole purpose of this video is to really give an introduction to it. So let's just start with a hypothesis. Let's say that you have a hypothesis that some pill that is made up of the petals of some flower, that this pill right over here, it improves, it improves running, running speed. It improves running speed if someone were to take it. So the important thing of any hypothesis, it has to be testable and so what you do is you have to think well, how am I going to test it? Well, what you can do, so how are you going to test your hypothesis? At first, you might say, give the pill, so give the pill to some runners, to some runners and test their time, test their 100 meter time, test their 100 meter time before the pill, before the pill and after and you might say, hey, maybe if, I don't know, their times improve after, maybe my hypothesis is correct. Pause this video and see if you feel comfortable with this test right over here, this experiment. Well, actually, there's several problems with this experiment. How are you selecting these runners and if you give them the pill and their speed improves, did it truly improve because of the pill or did it improve because of some other thing that they are doing? Maybe they got new shoes or maybe their diet improved in some way or maybe they just had a psychological improvement. This is often known as the placebo effect. If people are taking something that they think will help them, it often will help them even if that thing is just an empty capsule or just a sugar tablet. So how do you avoid these types of errors? Well, what you could do is you can find runners and put them into two groups. So let's say this is one group right over here and then this is another group and what you would wanna do is you'd wanna go into the population of people and you would want to randomly select whether someone goes into one group or another group. Why random? Because if you don't randomly select, there's a chance that there might be some implicit bias that you might just happen to be picking people who maybe their running speed is on an upward trajectory and they just happen to go into the group that will eventually get the pill. So you randomly, randomly put them in those groups and what you wanna do is you'll have a control group and you'll have a group that gets your pill and so this group gets the pill, gets the pill and now you might be tempted for this group to say, oh, they don't get a pill and then after a few months of it and it should be the same amount of time, you say, hey, did this group's times improve over the 100 meters? How did that compare to this group? But be very careful. If this group gets the pill and this group gets nothing then the pill might be providing that placebo effect again just making people think they're getting something that's making 'em faster. It might actually be a self-fulfilling prophecy. So it's actually important that you also give these people a pill although this pill would just look like a pill so this would be just an empty, empty, empty pill that looks the same. Now, there's another idea when you're designing scientific experiments that it needs to be double blind. Let me write this down. Double blind. So as you could imagine, it implies that two things are blind here. So the first thing that needs to be blind is the people themselves should not know which group they're getting put into. They should not know which pill they are taking because obviously, if you put someone in this pill, in this group and you say, hey, you're in the control group, we're just gonna give you an empty pill, well, then the placebo effect might not be, it might not work. It's also important 'cause it's double blind that the people who are working with the runners, the people who are measuring them so that the researcher is right over here so I'll draw someone with a clipboard. So the researchers who are observing these people and maybe administering the pill and telling them about the experiment that they too do not know which group they are administering it to because if they did, they might be able to signal somehow. They might be able to even subconsciously give a sense of which group folks are in and so let's say we do all of these things and so we're getting in the direction of a well-designed experiment and we find that the 10 people in this group versus the 10 people in this group after three months, these folks had a 5% improvement in running speed and these people had a 10% improvement in running speed. Is that enough to conclude that our hypothesis is correct? Well, you might be tempted, it seems suggestive but that's where statistics come into order because there's just some random chance that you got lucky, that you happened to pick the people that, your pill does nothing but you just happened to pick the people who are going to improve more and there's a whole field of inferential statistics when you take a statistics course that will go in more depth into this but essentially, what you're gonna do is you're gonna say, hey, assume that your pill does nothing, what's the probability of getting this result for 10 people or what's the probability of getting this difference in result and if that probability is very low, well, you say, hey, that would suggest that my pill actually does do something. Now, another important principle of an experiment like this is it needs to be replicable, replicable because even though you thought you did a good job, people might not wanna take your word for it and it's important in science for people to be skeptical. When people do experiments, they want to have a result and that bias might creep in and so if someone else does an experiment, you need to say how you did that experiment so other people can see if they get the same results because even though you think you randomly selected, you might only do it with people from a certain country or under certain weather conditions or assuming other constraints and then these people might do it slightly differently or in a different country or under different constraints and realize that hey, the explanation for this maybe was something else. Another thing to keep in mind is the larger your samples right over here, the larger groups that you're able to do this with, the stronger that the statistics actually become and I would say not just larger but the more diverse across genders, across ethnicities, across geographies. So the big picture here is all scientific progress is based on us designing good experiments and being very rigorous about how we think about those experiments and what I've highlighted here is just the beginning of how we might think about designing those experiments and as you go into your scientific careers, look at other people's experiments and see whether they've done these things because many times you will find that it is not as rigorous as it might seem at first.