Authors@Google: Salman Khan

>>presenter: Good afternoon, everyone. Let's get started. It's an honor introducing, hosting Khan Academy today, here at Google. We have three wonderful guests here. Sal, who's the founder, and Shantanu, who is the President and COO, and Marcia, who is a software engineer of Khan Academy. I learned about Khan Academy about a year ago. I wasn't sure how, just, it also comes with a lot of comments, also like people appreciating. I wasn't sure just how this video is going to be theoretical as far as personal education is concerned. So I tried it myself and I guess the first thing that I noticed was that it almost seems that he was having more fun teaching me than I was having fun listening to him. [audience laughter] He is so passionate about teaching. And I intentionally chose a non-technical topic, which I always hated, like history. And I watched through all the videos and it was just like glass and ice. At a very certain level, I realized why it works. With that, I will actually give it to Chris Uhlik. He will formally introduce Sal. He has been working with Khan Academy over the past year and he has actually helped them get a $50,000 grant from Microsoft Research and he also helped them get two million dollars from Google, as part of the 10 to the 100th Project. So, with that, I'll give it to Chris. [applause] >>Chris Uhlik: Hey, so, I have four children, ages eight through fifteen, and we homeschool them. My wife is very interested in education technology. She has a degree in education technology and it's illegal to experiment on humans, so she made her own. [laughter] And we've been basically educating our kids using 100 percent online materials. Every kid's got a computer; they spend all day at the computers and then they go off to physical education classes outside the home and stuff like that. So, she's a great student of what kind of education technologies are out there in the world. We buy everything; she researches everything well. Just over two years ago, she sent me a link to this Khan Academy. She said, "Hey, check this out. This one's pretty cool and itís free." And I started watching a few videos. I watched the banking crisis explanation videos. And I was like, "This stuff is really, really good." And I had been thinking about what are a few of the really big, high impact projects we can do in the world, like fixing carbon dioxide emissions, educating the people in the world. These are some of the huge problems where you can really totally transform the future. And I firmly believe that doing this for free, so that everybody in the world has access to education materials, by taking advantage of the leverage that technology offers, the ability to take one lecture given by one guy and then have a 100,000 people watch it over the next few months. And Sal's done 18 hundred little capsule lectures and they have a typical number of views--around a hundred thousand each--40 thousand here, a million there, give or take. So, he's reaching, right now, about two million people a day. >>Salman Khan: A month, unique a month. >>Chris: Unique a month. The University of California has .2 million, right? So, he's reaching ten times the University of California with his own efforts, a few people helping him, and a free video hosting service, right? So that's the power of technologies like YouTube and AppEngine to let people build incredibly impactful, valuable things for free. So, a year ago, I was working on this education project. It wasn't Microsoft Research that gave him the 50K, by the way. It was Google Research. [laughter] And I also helped the Google 10 to the 100 team decide and figure out how to give them one fifth of the total prize, which is two million dollars. And I think he's gotten some significant backing from the Gates Foundation and from John and Ann Doerr, and a few others. So, he's able now to really commit full-time and actually hire people and expand and he's getting some serious publicity help. That publicity has caused him to grow a factor of 50 in the past year. He was getting something like, 40,000 views a month a year ago; he's getting two million a month right now. And that's just incredible and it's probably going to continue accelerating at that kind of pace for a while. You can look forward to most people on the planet having seen a Khan Academy video in the next couple of years. That's impact. So, we're gonna start the talk. There's a Dory page at go/khanacademy. There's gonna be a lot of kind of question and answer period at the end. There's a microphone in the middle. Go up and ask your questions at the end. Don't interject and I'd like to introduce the Linus Torvalds of education. >> Salman Khan: [laughs] [audience laughter] He's gonna transform the operating system of the education future. Salman Khan. [laughter] >>Salman Khan: Thank you, thank you. [applause] Actually ñ I don't want to be -- how many of you all saw the Ted talk that came out a week or so ago? Ok, I don't wanna bore y'all, but some of y'all haven't seen it, but there will be a little bit of an overlap. And since you don't know the structure of this, I'm gonna do a quick overview and feel free to interrupt me at any time, or ask a question. And then we'll just open up to Q&A cause that'll be more fun. We'll learn things. But just to start off, as Chris mentioned, Khan Academy started off me making videos. So, I'll show yíall a little bit of a video montage to see a feel for what the videos are like, and we'll show you what else we're up to. [pause] [video plays] >>Salman Khan:(speaking on video) So, the hypotenuse is going to be five. This animal's fossils are only found in this area of South America; nice, clean band here and this part of Africa. We can integrate over the surface, and the notation usually is a capitol sigma. National Assembly that create The Committee of Public Safety, which sounds like a very nice committee. Notice, this is an aldehyde and it's an alcohol. Start differentiating into effector and memory cells. A galaxy. Hey, there's another galaxy. Oh, look, there's another galaxy. And for dollars is their 30 million plus the 20 million dollars from the American manufacturer. If this does not blow your mind, then you have no emotion. [laughter] [end video clip] >>Salman Khan: So, that's a feel for what we're doing. And as Chris mentioned, we are now -- there's now on the order of 2200 videos. I made five this morning. And they're-- [laughter] ten minutes, you know. And they now cover everything from basic arithmetic all the way to vector calculus and the French Revolution, and all the rest. And we're reaching -- actually a month ago, we were reaching a million unique students a month. Now, weíre reaching two million unique students a month just with the latest buzz. So that's kind of where we are now, but there's a lot more that we're up to. Yíall have very generously given us two million dollars, so I view this as a progress report. [laughter] But before I go into that, I'll talk a little bit about how I got started. And as yíall know, I was a hedge fund analyst five years ago in Boston. Then, we moved the firm out here in Northern California. It was actually a two-person hedge fund and my boss -- his wife became a professor at Stanford, so he moved to Palo Alto. And I was tutoring my cousins remotely in New Orleans and started working with one cousin, then another cousin, then another cousin. Before I knew it, I had this cohort of cousins all over the country and I was looking for a way to scale myself up. And one of my buddies [Sulfaka Ramzana]--I should give him credit--I was literally hanging out at his house, showing him how I was tutoring my cousins and all that, and he's like, "You know, why don't you put some of your lectures on YouTube?" And I said, "Oh, that's silly. YouTube's for dogs on skateboards. It's not for-- [laughter] It's not for a serious learning. But once I got over the idea that it wasn't my idea, I-- [laughter] said, "I'll take a shot at it." And I remember the first video, I think, and you can go there, literally, it's like November 6th, 2006. It was greatest common divisor, least common divisor. One of those. I wanted to teach my cousins fractions. I was like, "Oh what do they have to know? And the negative numbers." And I put in like, 20 or 30 videos and this has turned into a bit of a one-liner, but it's true. The very first feedback my cousins gave me were that they preferred me on YouTube than in person. And so, I felt like this was something to do. So I kept making the videos. I started getting feedback from people all over the country, saying how it helped them. "Hey, this helped me on my exam. I passed the exam." But some of them were like, "Hey, I was gonna drop out of high school until these videos". Or "This motivates me to go to college and become an engineer." Or "This is the only reason why I can, now that I'm retiring from the military, I can feel comfortable going back to the community college." So, I was excited. So I kept going. And along the way, the site grew and then the other thing that happened--actually this is what most people don't realize-- sometimes, I switch around the story because it sounds better when I started the videos. But I actually started on the software side. If you rewind before I even wrote that first piece ñ I recorded that first video, when I was tutoring my cousins, I would just point them to random websites. I was like, "Hey, I just found some website run by this university and there's ten good problems on fractions. Why don't you do those problems and we'll go over them tomorrow?" And the next day, I'd say, "Hey, now you do the problems?" She'd be like, "Yeah, yeah, I did them." I was like, "How many did you get right?" She said, "Yeah, I think I got them all right." I was like, "Oh, when did you do the problems?" "Oh, yeah, I did 'em like at night." [laughter] It wasn't that ñ it wasn't that informative. So I was like, "Oh, I'm gonna write my own problems." And then she would run out of problems. There's these little worksheets that you see all over the web, like ten problems and then you're done. So I wrote these really primitive JavaScript generate problems, as many as you want, in adding fractions, or adding negative numbers, or multiplying fractions, or whatever. And so that was -- the premise was, "I'll give you as many problems as you need until you get ten in a row" and I'll show you what they look like in a second, "and you'll get hints for them." And before I even made the first video, I thought that was the solution. That was like, "this is all someone needs to do math." Because on the exercises you have hints and all of that. But then, once the videos took off, it was just a better use of my time to make five videos a day rather than one module every five days. So, that kind of ñ it got orphaned a little bit. And actually, one thing that happened is, I left it out there and I had this 50 dollar a month webhosting and at some point, there were too many people using that software, so I just turned it off -- which is probably a bad idea if you're ever starting a business. [laughter] It's probably a signal that you shouldn't. Maybe spend a hundred dollars on your webhosting. So when the opportunity -- once Khan Academy started growing, we had this viewership and it seemed like there was an organization that we could start here. The question was like, "How do you take what we're doing to the next level?" And that's what ñ [pause] And that's what I wanna show you right here. So, these are the exercises and what I started with my cousin was a much more primitive version, but this is the same, actually some of the same basic code. It's been fancied up a good bit now. But the general principle is, it'll give you as many--this is subtraction one--it'll give you as many problems as you need until you get ten in a row. And it's a very simple--it got cut off here-- but you can have the videos here, there's hints, you can see it draws a number line for you. The Khan Academy videos can be pumped in, and itís a very simple idea; you do it until you get ten in a row, but itís--at least in our minds--completely different than what happens right now in a traditional school. In a traditional school, lecture, homework, lecture, homework, lecture, homework, lecture, homework, then snapshot exam. And regardless of whether you get 80 percent, 90 percent, or 95 percent, the whole class moves on. Even if you fail an exam, actually, the whole class will move to the next topic and what was that five, twenty, or thirty percent you didn't know? It was probably something that you actually probably need for the next topic, or definitely something you'll need in a few years. And there's no -- people just ignore it. They place a label on your head; that some kind of value judgment on whether you're smart or not and then everyone just proceeds down with those gaps. And so, what we're saying is, "No, we're gonna have them do the opposite." Instead on penalizing you for failure and not expecting mastery, we wanna do the opposite. We wanna allow you to experiment. If you're learning ñ if you're learning to ride a bicycle, you would just sit on that bicycle as long as it takes to actually learn how to ride the bicycle. And so, that's what we wanna do here. We wanna allow you to experiment, we wanna allow you to fail, but you're not going to move on to more advanced topics until you actually, until you actually master the topic. So this is -- that is the subtraction one module. This, right here, is trigonometry. [pause] This, right here, is shifting and reflecting functions. And then this is how all of the exercises are structured. So literally, this node right up there is single-digit addition. It's literally five plus seven, or one plus one. And once you get ten in a row there, it'll move you up to one digit and a double digit addition, and then single digit subtraction and then we call it very basic multiplication. Once you get ten in a row there, it keeps moving you down what we call this "knowledge map" and if you keep going down the knowledge map, you start getting into some more advanced arithmetic; some pre-algebra here. Go further down and it starts getting into algebra, a little bit of trigonometry, geometry, and precalculus. And so, the general idea is that we want this graph to eventually cover everything. Actually, the funding that yíall have given us, roughly half the funding is to translate the videos into ten languages, but the other half of the funding is to build out this graph structure to cover on the order of about two hundred or three hundred modules. So, literally is all of K through calculus mathematics. But the goal, and it's already pretty good coverage for K through nine, or K through ten mathematics, but the goal is right now, it's about 107 modules. The K through calculus is probably gonna be on the order of about 300 modules; totally very comprehensive. But there's no reason why you can't expand from this into logic, into computer programming, into genetics, into probability, into finance, into accounting, into grammar and logic. You can just keep building off of this knowledge map, so it really covers everything that can be quizzed in this type of a form factor. [pause] And so, this is just some more of it right over here. And so the paradigm that we're doing -- when I started this, I assumed it would be a kind of a nice to have -- a supplement for people. Even when we started, when we formally became an organization, we didn't assume that it would be adopted in schools. But then, one thing that happened was that teachers started adopting it on their own; just the videos. We started getting letters from teachers saying, "Hey, you've already given the lecture, so we're using those to flip the classroom. So instead of me giving the lecture ñ me the teacher -- , I'm assigning your lectures as homework. And then what used to be homework, I'm having the students do in the class." And it's a very simple concept, but it really changes what a class is all about then. Now, all of a sudden, instead of you literally have 30 people completely silent and completely passive, most of them zoned out, a teacher having to give this one size fits all lecture, even a great teacher, they're losing probably two-thirds of the class, now that happens at home. You don't have to be embarrassed to rewind and look at something that you might have missed, or fast-forward if you're bored, or pause something ten times; you don't have to interrupt the whole class. And now, when you actually go to class, you actually have all of your peers, you actually have the teacher to actually help you out. And it's interesting. One thing, and I had mentioned this is in the Ted Talk, "that'll work for motivated students, but what about the students who aren't going to do that?" And I was like, "Well, if you're not gonna do anything at home, period, it's still better that you're doing the exercises in the classroom, because that's, frankly, where you're gonna get most of the learning in and if you didn't do it, watch the video in the classroom, too. And so the paradigm is, where you really learn stuff and where you're really getting your head around something, you want other people to be around you. When you're actually trying to solve the problem. But when you're trying to listen to a lecture, you don't want people around you. You don't want your peer to say, "Oh my God, look how stupid Sal is. I didn't have to pause that. He's reviewing stuff from 30 -- " You don't want that around. So you want the lecture to be intimate, but you want the actual classroom experience to be social. So what we're doing in classrooms now is taking that to the next level. Los Altos, right here, they came to us, they came to us actually about four months ago. Shantanu and myself, we had a meeting with their school board and said, "If you just had carte blanche in a classroom, how would you run the classroom?" And we said, "Well, we would let every student work at their own pace on those exercises on that knowledge map we just showed you, and the teacher would walk in every day and just get a dashboard that looks something like this." And this is actually the live dashboard now from Los Altos, just so you know how the story ends. "And the teacher would let -- everyone would work at their own pace and only intervene on the kids who were stuck." And they said, "Oh, that's a very radical approach." But then, the next day they said, "Oh, we'll do it." So, Los Altos, in November, they started with two fifth grade classrooms and two seventh grade classrooms, and this became ñ I would say this became 50 to 60 percent of their math curriculum, with the other forty percent being project-based learning. So they completely gutted the traditional part of the curriculum; no more lectures, no more traditional homework, no more of the lecture-homework-lecture-homework cycle. Now, the students walk into class, the teacher looks at a dashboard like this. Each row, here, is a student. I've blanked them out for privacy. Each column, here, is one of those concepts you saw on that knowledge map, and green means the students already got ten in a row, blue means that they're working on it, but no need to worry, and red means that the student's stuck. So there could be a kid working on third grade math in a fifth grade classroom, and there is. That's cool, as long as he or she isn't stuck. There could be a kid working on trigonometry in the fifth grade classroom, and that actually is happening. And it's actually entertaining because a reporter who asked the girl, she was literally doing trigonometry and she's a fifth grader and she's like, "Do you think this is fifth grade math?" And she's like, "No. I think it's sixth grade." [laughter] But that's cool. They should both be working on those respective concepts because that's where they are. But, the teacher just works on whoever is stuck, or even better, get one of the kids who are already proficient in that module, who've already gotten ten in a row, let them be the first line of attack and actually tutor their peers. And we all know when you actually tutor someone, there's a whole other level of learning that's going to happen when you do that. And the teacher only intervened -- and so, every moment of a teacher's time is actually spent on being a human; being ñ kind of interacting with the classroom. I've mentioned this in other talks is that whenever people talk about technology, they assume its like, "OK, it's probably good for economics. It's probably good to get more bang for your buck, but it's probably gonna dehumanize the classroom." And the one thing we keep saying is, "No, itís the complete opposite. When you do something like this in a traditional classroom, only about five percent of a teacher's time is actually being a human being, actually sitting down next to someone and tutoring them, mentoring them, or working with them. Now, a hundred percent of their time is." So, when you're using technology, you're actually humanizing the classroom by, if you believe those numbers and I actually think the five percent was generous, by a factor of 20. And so, we don't want, even when I worked with my cousin, Nadia, and even when I had the data on ---, I would ask her, "Did you get this? Did you understand this?" And she's like, "Oh, yeah, I got it. I got it." There's these very painful discussions that you have if you ever tried tutoring anybody on what they do get and what they don't get. And so, we wanted to avoid the teacher having to do those really painful, awkward questions. And so, we're trying to arm the teachers with as much data as possible. So this is like ñ this is one of the views. A student can view this about him or herself, or a teacher or parent can see this about any of their students. And this is just one of the reports; we have a bunch of them. But this is literally just the activity for the student over the last 30 days. You could see it over the last day, you could see it by hour, you can see it over the last year. But each of these bars, these are on a certain day. The light blue is the time spent on exercises. The navy blue is the time spent on videos. These are the highest achievements achieved that day. And if the student or the teacher, they scroll over any of this, they actually see exactly the student spent 38 minutes on exercises; they got that many energy points. These are their achievements. The videos would be there if they watched videos that day, and we're actually even tracking when they pause the videos, when do they repeat the videos. We're gonna start using that. One, for the teacher to know what happened, but also even for us to know how can we, how can we improve our videos. [pause] One report that teachers ask us is -- they said, "Look, it looks like some of the kids are really focused and they're moving ahead, while some kids are jumping around a lot. Can we have a report on focus?" And we said, "Sure." So, this is, this is the focus report. The outer circle is literally a pie chart of what exercises students spent time on. This inner circle is where they spent on videos. And, once again, if you scroll over any of it, you get more detail on exactly what the student was up to. [pause] This right here just shows how granular the data is. This is a student who is doing the probability one module. So, a teacher can go and say, "OK, this is the first problem; they got it right. That's why it's blue. They probably guessed because they spent like, four seconds on it. And they watched the video. Then they got the next one wrong, a couple right, maybe guessed. Wrong, wrong. It looks like whenever they actually made an effort on the problem, they got it wrong and then they guessed a few right. [laughter] Then they used a hint. And if you click, if a teacher actually, if a student, later, when they see this on their report and they click on any of these, they see the exact problem that they didn't get right. And then, you can eventually see-- and this was so funny--it got clipped off here, but you can see the student eventually getting ten in a row. And it's fun; you can almost see them thinking. There's some deep thought going on the 99th problem, where the student said, "No, I'm gonna get my streak now." And so, he or she literally got this problem right, and then you can actually see that they get faster and faster and then they get ten in row. So, you can actually see the learning process. [pause] And most people ñ when you say "self-paced learning" sounds like a good idea, but the one thing that we're seeing over and over again. And this goes back to ñ I -- there's literally in one of the fifth grade classrooms, in Los Altos, there's a few kids doing third grade math and there's a few kids doing even calculus. And there's a huge temptation when you see that. "Oh. Gifted kids, remedial kids. But it's cool. They can be in the same classroom, but those kids are gifted, those kids are remedial. Those kids are not gonna be at Google, these kids are. I mean that's the -- we all do that. When you see that, you just think that. We're so used to labeling people. But one thing we're seeing over and over again, we've seen it at Los Altos, we're seeing it in other summer camps we've run, is that some subset of those kids--five days into it--there's this spread. This is just days into the pilot. This is the modules completed. Some days into it, there's a group that clusters here and you can see that group here and there's a group that clusters here. These are the gifted students in a traditional system; these would be the slow students. But you always see these students, once they spend extra time on some module, and they get some core weakness out of the way, and you don't know who they are ñ you really don't ñ you have no idea; they just rocket forward. Everything else just starts to click and so, in a matter of six weeks, we've seen this flipping multiple times of the students that you used to think were slow or remedial; they just had some basic core weakness. And once you actually give them a change to get it out of the way, they just rocket ahead. I mean, this kid right here, literally, I think he's our number one or number two kid in the class, and starting off, he was one of the worst. So you see this over and over and over again. [pause] So that's the general idea of what we're up to. We can address a lot more of it in the Q&A, but our goal is literally to be a free classroom for the world; give a world-class education to anyone, anywhere. And I want to be careful. We don't think we're doing that yet. We think we're solving parts of the problem. We have the videos, we're doing the exercises, we're doing the data, we're doing the analytics, we can kind of become a kind of operating system for a classroom, but we think there's a lot more and that's why itís fun to talk to yíall because to some degree, we can even brainstorm on what that would mean. But, I think it's completely possible that you can have a reality where, in five or ten years, ideally you will have a classroom where you will have a human being actually helping you out. But there's a reality where even if you don't have that, you can get a pretty darn good education. So I'll open it up to questions. Anyone who wants to walk up to the microphone, and if yíall, I guess there's that website where people have ranked questions. [pause] I could ñ I'll take this one. >>audience #1: Hi. Thank you for coming. I wanted to ask a question about if the model for the future is gonna be one like Wikipedia, where you're gonna open up the creation of the content, the videos, and the exercises? And if so, how would you maintain the quality in those types of things? >>Salman Khan: Yeah. That's a great question. The question is, will we open it up to kind of be a Wikipedia of education as opposed to -- and so, we're open to opening it up somewhat, but I think for video-based lectures, if we just look at the video side, I think the reason why I think that kind of crowd sourcing doesn't work is because it's one, your quality is gonna be all over the place. And even if the quality is good, it would actually be very jarring. If I came and give you -- imagine if you're in a math class and I came and gave the first lecture, and then the next day you have another good lecturer, but you're like, "Who is this guy?" And then the next day you have another lecturer. So there's actually this huge level of trust that you have to build with the lecturer for you to be able to invest in more and more watching. So, that's why we resist really opening it up. We do suspect there's a whole set of things that other people know that I could never know, or that would teach better than me. Or maybe they'd teach the same thing that I'm teaching, but in a different way that responds to other people. And so we are open to people. We're telling people, "Hey, make 30 videos, 40 videos, 50 videos on a certain topic." One, you should do it on your own, cause you don't need us to broadcast them, but if it really does feel the same, it's consistent with our brand, we'd be open to it. >>audience #1: Thank you. >>Salman Khan: Yeah. I'll take-- >>presenter: [ ] >>Salman Khan: Top down. OK. I'll read this first one. "With such busy schedules and varying priorities, how are parents/students supposed to identify online learning centers like Khan Academy that are legitimate academic institutes and not just money-making schemes created to take advantage of the online trend?" One, we're not-for-profit, so we're definitely not a money-making scheme, unless I got particularly creative with the-- [laughter] But we're not a money-making scheme. And actually, that was, that was one of the main reasons for being a not-for-profit. I mean, there's a lot of people who talk about double bottom line organizations which is an oxymoron. Bottom line is meant to be literally the bottom line. Like that's the thing that you care about. And you can't say I care about two things that are usually in opposition with each other. So, if I was genuine, and I hope I am, about -- no, the number one goal is not to become rich, not to do whatever. The number one goal is to provide an education for people; make it a not-for-profit. And there's other realities that it actually has a better chance of becoming a lasting institution then. If it's for-profit, it'll be acquired by McGraw Hill at some point and I'd buy a nice house, but that would be the end of it. But here, there actually is a potential to last potentially beyond me. I'll take a question. >>audience #2: Yes. So one of the problems I have is -- we run a training for our Google partners -- it's technical training. We have a classroom-based and it's like your model. We focus on getting the learning exercises; walking around, helping them, get through them covering the theory. And we have another group of you who are ñ that can't come here for various reasons. >>Salman Khan: Yeah. Yeah. >>audience #2: I have problems getting them to actually do the homework because that's where the learning, watch the tutorials, but the homework pieces seems kind of ñ it drops off. People are not that interested. What kinds of things can you recommend? >>Salman Khan: You know ñ I think ñ so -- to get people to do the homework, I mean -- the best way is just to track it. I mean, that gives you information on whether they are or aren't doing the homework. >>audience #2: They're not submitting it, so I know-- >>Salman Khan: Yeah, yeah. So, they're not submitting it, so they're not doing it. >>audience #2: And we have some carrots. We have certifications and that's a big carrot that we're seeing people completing the content. But are there other, because people ñ there are competing events they have in their life. >>Salman Khan: I think -- it's interesting we had the gut sense that gamefying a lot of it, giving badges and points, and actually I can even show you we have some of that over here. We have some of these ñ we have some of these badges right over here. So, gamefying, we thought it would be kind of a nice to have, like it would make it kind of cool, but itís amazing what's itís done. Oh -- it's not on the screen now. >>audience #2: Have you seen a lot more people completing the content with the badges? >>Salman Khan: Yeah. It's great just changing the amount of points you give for certain achievements or the wording has actually changed like thousands of people's behaviors overnight. Oh, yeah. There you go. Those are the badges. So we just have a ton of these badges that we've set up and I was saying earlier at lunch, in Los Altos, these are fifth graders. So, lemme show you there's a black hole badge that right now, the black hole badge is -- we have it defined. It's the most legendary badge on the Khan Academy. It's actually impossible to get right now. [laughter] Because we said no. That has to be -- once we have three hundred modules, then you can only, if you get like, two hundred, we don't even have that, so itís actually impossible to get now. But the fifth graders became so obsessed with getting--fifth gradersó with getting a black hole badge that some subset of them went into our code, so itís an OpenSource project, [laughter] went into our code and figured out they had to get 150, they told us, "Oh, we figured out we have to get 150," and I'm like, "How do you?" "The code." [laughter] So, itís amazing what, I mean, yíall should be giving offers to those kids. I don't know what they're -- [laughter] But itís amazing what this type of thing does to motivate, especially kids. I don't know if it'll do it for employees. [laughter] >>audience #2: Thank you. >>Salman Khan: Yeah. Those, exactly. And I'll take one from the -- I'll just read it. Where's the? [pause] Oh, there's not a, oh, these are always the same. OK. "So you tend to focus on math, science, and economic concepts. What do you think it would take to make this format work for grammar, writing, or composition?" So grammar, I think, is one that's very doable. I've tried to contact people who have good--I won't name the companies--but there's a company out there that I think has a really good, you know -- when I studied for the GMAT, I used their book, and I was like, "I finally learned grammar." [laughter] And it might not be obvious, hearing me speak now, but I just wanna find good materials. But, I think, grammar actually is perfect for both the videos and for the exercises, because it is actually logical and very mathematical to a large degree. And you can diagram things out. I think, one way or the other, writing or composition, those will be interesting. I think writing and composition are much more activity-based. You can't just have someone give you a lecture on what it means. Although, it would be probably ñ it would be interesting to have someone give a lecture. Maybe itís me, maybe it's someone else. On what -- like, this is well-written, this is badly written, and why it is. But the more useful thing is to actually to write things and maybe get some type of peer review. So, it probably won't fit directly into that knowledge map structure, but I think there probably, there is something we could do to leverage the two million students using Khan Academy right now, where you write an essay -- maybe we have essay competitions, or people write essays and then they peer-review it and the best ones surface. I think there's something that we could do, but we don't -- we haven't figured that out yet. I'll take one right here. >>audience #3: So I'm a huge fan. I actually watched most of your linear algebra lectures over the winter break, and I've been trying to tell everyone I know-- >>Salman Khan: You deserve a badge. [laughter] >>audience #3: I thought about it; different things to get badges and showing them to everyone. So I've been telling everyone I know about Khan Academy, but I was brainstorming with other engineers here about how we can do -- what can we do to help? As you mentioned, other people giving lectures isn't necessarily helpful because they might not be good lectures. So, what can other individuals, or Google, do to accelerate this process other than just giving money? >>Salman Khan: Yeah. Oh you could give more money. [laughter} >>audience #3: Well, I mean-- >>Salman Khan: No, no, actually, genuinely, geniuinely and you're right. I mean, we view the ñ actually the videos are kind of what get people into our fold, but then the real value is the exercises and the other. And I think, eventually, the communities are gonna be super valuable. So, I don't know the policies of volunteering code for other projects and things, but those modules, they're very stand-alone. We're speccing them out and if you say, "Hey! I could write a cool or I could write a whole set of modules on grammar." We'd love that. Work with us; we're literally based on Castro Street, so we're not too far away. But if there's more sophisticated things, people at Google, even outside of the funding, have helped us move forward with some of the translations and things like that. So, there's interesting things out there, AdGoogle, either products that could directly benefit some of what I've talked about, or you're like, "Hey, I think it'd be fun to crank out a couple of those modules or have some neat ideas for them", we'd love to have your help and we are hiring, if any of yíall are looking for a sabbatical. [laughter] I don't know what the policy is on things like that. I'll -- >>audience #3: I had one more quick question, 'cause-- >>Salman Khan: Oh yeah, yeah, sure. >>audience #3: How much planning do you do for your videos? >>Salman Khan: How much what? >>audience #3: How much pre-planning do you do? >>Salman Khan: How much pre-planning do I do before the videos? So, if I'm doing like an algebra worked example, I literally will like, put it up there and just do the problem. Like the first -- when you see me do it, it's usually the first time I'm looking at the problem, unless something disastrous happens. If I'm doing, like I did some videos recently on China. Actually, there's one of the ones on the montage on how China pegs its currency to the US dollar; that one is something I'm kind of familiar with from my hedge fund life, so on my walk to work, I'll think about it and then when I sit down I'll just do it. So, if I'm doing something that I'm really not that familiar with, where I really have to get myself re-immersed, like organic chemistry, that I'll spend like two weeks literally just immersing myself in the field, and getting the ñ kind of living and breathing organic chemistry. And then I'll make the video. I tell people that I try to prepare my brain, but I don't try to script the lectures, because then they come off unnatural, or scripted. [laughs] I'll alternate. How can we be sure that great programs like this don't widen the gap between the haves and have nots? Those who need the most help are often unlikely to have a home computer or a high speed connection. So, that's actually a really good point, and I suspect, it's not something that I'd like to admit, but I suspect that right now we maybe are; that right now it is people, I think Khan Academy's penetration is much higher in Google households than it is in random households across the country. And I think itís definitely much higher than households where there isn't a parent with a college education. So I think right now we are, probably in the short term, unfortunately doing that. I think over the long term, though, what it does is it--right now when everyone says, "Oh, we should get one laptop per child or we should put fiber to this school or that school." There's not a tangible reason to do it. Like people -- there's a feel-good reason that maybe it'll help. It'll probably help kids have the access, but I think once something like a Khan Academy exists and people see that ñ we better get our act together -- this will actually be the catalyst for getting fiber into the classroom, the catalyst for getting laptops in front of kids. Then it'll actually close the gap because the delivery cost is actually zero. One thing that we're saying is -- Los Altos is not a needy neighborhood, but they were quick to adopt it because they're flexible and we were really impressed with how they run. There's some private schools that want to adopt it really fast and there's no bureaucracy or nothing, so they're adopting it. But what we suspect is going to happen is they're setting the best practice and then, hopefully, around the rest of the country or around the world, all sorts of upper middle class, middle class, lower middle class, will say, "Wait, if they're doing that at Los Altos, or they're doing it at that fancy private school, how come our kids are not doing it? And on top of everything, it's free. So, there's no reason not to do it." So I think in the near term, there might be a little bit of that, but hopefully over the long term, it actually serves as a catalyst to close, to actually motivate the technology gap. >>audience #4: Hi. My name is Anne, and I should admit that both of my parents are math teachers and I was a math teacher before I came to Google. So, it's really exciting to see, like, I've certainly seen a lot of times with my family being skeptical of various ways of technology trying to replace, but it's really exciting because it's very much correlated with the types of challenges. And my mom is especially into self-paced learning, which is really time intensive and people-intensive so it's a really amazing way to let that happen and let the teachers focus on the most important place where they can add the most value. I was wondering how you've seen teachers incorporate this in terms of when they want to do deeper problem solving, or group work, and how does that, how does that classroom experience integrate and -- >>Salman Khan: Yeah, so the question is about how does this integrate with what teachers are trying to do with a deeper experience and we haven't, I get a lot of anecdotal stuff on emails and stuff and I said, even before we started doing this pilot with Los Altos, there were teachers who are flipping the classroom, lectures at home, or home and in the classroom, and then the teacher can work with them. And what we're seeing in Los Altos is, and even those teachers, is because you're freeing up all of this time where this lecture doesn't have to happen in the classroom, it depends on the teacher, but it frees up a ton of time so they can do the project-based learning. There's so much debate: project-based learning versus the traditional doing a lot of exercises and seeing a lot of problems is learning. And what's cool about this is you can do both. They're both valuable. It's actually not an either/or proposition. And even a lot of people said, "Do you teach new math or old math?" I do both. You know, why not? There's no reason ñ there's no limitation for our classroom, so you can actually do every possible way of looking at something. >>audience #4: Awesome. Thanks again. It's really amazing. >>Salman Khan: Yeah. Oh, thanks, thanks. I'll take one here. Students are now truly--I'm doing it like the videos, I don't know how bad these questions are gonna be ñ they might put me on the spot-- [laughter] "Students are now truly empowered to work through course material at their own pace. How do you create meaningful cohort groupings in this context? Age--as opposed to skill--seems to be inefficient from the perspective of maximizing learning." So, I completely agree with that. So this, even in the Los Altos classroom, you have--it's a fifth grade classroom; they're all ten or eleven years old--and there you have that mix. Our idea would actually be a classroom of all age groups. Literally, in the same room, you have fifth graders, or you have five year-olds and you have 18 year-olds, and there's no kind of preconceived notion that if you're 13 and you're learning this, you're stupid. It's just the pace that everyone learns at. And what you get there, I actually think everyone in the mix will actually act a lot more mature when they're around older and younger students. And then you get the true benefit of the learning across generations. So, I completely ñ I completely agree with that; that it's a good comment, I guess. [audience member claps] Yeah, yeah, yeah, yeah. I'll take-- >>audience #5: [ ] >>Salman Khan: Oh, very good. Thank you. Go ahead. >>audience #6: Great. After all these very wonderful questions, I have a really pragmatic one. >>Salman Khan: Yeah, yeah. >>audience #6: Which is, your lectures, for example, on the French Revolution and the California High School Exit Exam, use imagery and problems from other sources. How do you dance around copyright? >>Salman Khan: How do I dance around copyright? So I have a very loose interpretation of "fair use". [laughter] No, no, no. Well, that's actually true. But the -- so, all of this, like the French Revolution stuff, I do look on Wikipedia, I look for stuff that's in the public domain. Most of those paintings and stuff are in the public domain. The copyrights expired. Stuff like the California or the Massachusettsís Test for Teacher Licensing, I mean, these tests are written by government bodies and I assume they want teachers to pass. And so, I've taken--on things like that, if it's a government sponsored-- I also have stuff for like the IIT Joint Entrance Exam, I have some of their problems. These are issued by the Indian government. I didn't ask their permission. But I assume, I mean, they can sue me. It would be an interesting PR issue for them. [laughter] But I ñ I ñ I ñ one -- I actually do think it falls under fair use, but even -- I would be very surprised if they were upset. I wouldn't take something from McGraw Hill and do it, is a simple answer, or I'd ask their permission first. >>audience #6: Thanks. >>Salman Khan: And I am careful about the Wikipedia creative commons and all that other stuff, too. Yeah. Simple. So, next one." What tools and techniques are you using to interest students who are traditionally not self-motivated or those requiring a different learning style than a standard video provides?" So, traditionally not self-motivated. Actually, the gaming is a huge thing for the traditionally not self-motivated. And actually, I think once -- a lot of people are just not motivated because they have these gaps in their knowledge that the traditional model can't address and people are just talking past each other. And if you allow them to remediate, without feeling embarrassed, and they can do it in a game framework and they get points and badges, that, actually, I think solves a lot, a lot of the actual motivation problem. The other ñ the other -- the learning styles, what are you doing requiring different learning style that a standard video provides? So there's a couple of things. I mean, there's the video, but I think the exercises are at least as important and that's what we do more for the peers to be able to interact with each other. That's as least as important. But the other thing is--this isn't Khan Academy related--but they've actually done a lot of studies, like double-blind control studies and the learning styles is kind of a myth; that there's actually a good instruction and bad instruction. Everyone likes to think that "I'm an auditory learner, I'm a visual learner," but the reality is ñ no -- there's some things you learn, you're learning art history or geometry, you're a visual. If you're learning a foreign language, you're auditory, or maybe a little bit of visual. If you're learning certain things, you like to see it written. And when they actually did double-blind, this is easy -- you can go Google it. There's a bunch of styles; I was just reading this last week, that there's actually no evidence that to teach people who are self-identified, or somehow identified auditory learners in an auditory way, actually is better than not. But with that said is ñ with that said, I think there are multiple ways to teach the same thing and so, I'll make videos. I think in the future, we'll probably have other hopefully good lecturers who will take a slightly different take on things. Even some topics, I take five different takes on it, so people see it from a bunch of different angles. Maybe at different skill levels. Even now, there are some videos that are very primary, a little bit more mechanical, but some go a lot deeper and go into the proofs and go into the actual intuition. So I think itís just making sure that there's enough material out there that it responds to every type of learner. I'lló[pause] >>audience #7: The way, the implementation, that you've done is great, but the ideas aren't really new. >>Salman Khan: Nope. >>audience #7: They're 30, 40, 50, 200 years old. >>Salman Khan: Yep. >>audience #7: So, I was wondering how much, how much you've been able to tap into the small bit of this kind of learning that's been done in the past, talking about like Sudbury Schools-- >>Salman Khan: Yeah. >>audience #7: or some of the very radical educational groups, un-schoolers? How much has been able to flow back into the things you're doing in Khan Academy? >>Salman Khan: Yeah. So the general concepts here are not at all new and actually, I was recently looking, I mean people -- this whole notion of actually mastering something before you move to another concept, it's called mastery-based learning and people have been talking about it for 80 years and have actually done studies on it, like 30 years ago. The two definitive, in all of the education research, the two definitive studies, or at least the only two that I've found that are just super definitive, are one, mastery-based learning. Every trial they did, and they did this 30 years ago with like worksheets and people walking around, but it was essentially the same model had unambiguous success. I mean, these kids were learning at several fold the pace of traditional students, with much better retention and much better mastery of things. And then the other thing, there's actually a good bit of research, 30 years old, definitive, never refuted, is that when people watch a lecture, they zone out after 15 minutes. And they zone out for about five minutes and then when they zone back in, they're only able to zone in for about nine minutes. And then they zone out for another five minutes and then when they zone back in, they can only zone in for about six minutes. And this process goes until, by the end of a 90 minute lecture, they can only zone in for like three or four minutes. And I think we've all experienced that. Hopefully, none of yíall are right now. [laughter] But these are not ñ but to the question's point, these are not new ideas. Even when radio came out, people said, "This is gonna revolutionize education; get the best lecturers, put them on radio." TV came out; same idea. VCRs came out; same idea. I think what makes it different this time is it is so easy to get at. I mean, it is literally on demand, and you have all of the data, all of the analytics. And I think there is something about the actual craftsmanship. This wasn't built by a bureaucracy at the Department of Education, or a bureaucracy at a publishing company. This was built by people who are literally looking at the learner and saying, "Well, what does the learner want? And what's useful for them?" In terms of actually leveraging some of that stuff, I think we can leverage the spirit, I think we can leverage a lot of the models. And it gives us confidence that look, there's actually a huge amount of research, it's just nothing exists because it wasn't practical until now. And I think that's probably the level that we can leverage a lot of these other models. I'll take one over here. "Your program demonstrates the role that technology can play to make learning efficient and effective, yet the adoption rate for technology in the public education system is lagging. What hurdles must we overcome to get these tools in the classroom?" So I think the technology in the public education system is lagging for good reason. I don't know -- even when I was in school, in the 80s, we had computers in the classroom and we would go into the computer lab and we would learn how to insert a diskette. Or, really silly stuff, like how to boot the computer. There was no actually using the technology in the classroom. And I think that goes all the way now where people have iPad's but they're still not really doing something constructive. So, I think a lot of the skepticism was actually warranted. And hopefully, Khan Academy will be a catalyst for saying, "OK, finally we have something that is actually something to do with the technology in the classroom." And so, I'm hoping it's actually going to be a catalyst for solving that problem. Go. >>audience #8: I mean this is a very disruptive approach, things like flipping the classroom and so on. Are there any stakeholders where you've encountered strong resistance? I don't know, parents, principals, teachers, or somebody? >>Salman Khan: Yeah, so have you encountered any resistance? It's strange, but no. One, because from the get-go, you know the way -- it's been a very organic process. So, never, five years ago, even now, we're not saying, "We are going to lobby the Department of Education, or this state body, or this school district, and we're going to try to sell our product to them." If we did that, we would face a lot of resistance. What we're doing is we're literally putting stuff out there. We have mainly the student in mind and forward-thinking schools and institutions and school districts are coming to us and so, they've been, they've been pretty amazing about doing it. I think one interesting thing that'll happen, I think for almost everyone in this room, if you were to be a school teacher, this would be a more fun way to teach. You get to go into a room, you don't have to prepare. [laughter] And, and you have -- you get to be a mentor for these students. I think what could be a little scary for some teachers is that you're a fifth grade teacher, but there's some kids doing calculus in that room. And what we've seen with the Los Altos students, it's actually pretty profound what they've been doing. They walk up to those kids, cause some of those kids go red on like, the chain rule, right? [laughter] And so, the teacher's like, "I have to intervene." [laughter] She'll walk up to that student, she's like, "You know what? I haven't seen this since college and I forgot how to do it. Let's learn about this together." And it takes a lot of self-confidence for a teacher to admit that they aren't the know-it-all at the front of the classroom, but they are just maybe a more mature, someone who's seen more of the world, but they're at the same level mathematically right now but they can learn it together. So I think that's the paradigm shift that will have to go on, but I think it becomes much more pleasant as a teacher to be able to do that. And the kid immediately responds to the teacher. The kid immediately respects the teacher more when he or she actually communicates in that kind of way. >>audience #7: And parents seem to be OK with it, also? >>Salman Khan: Yeah, with the Los Altos, we had no experience actually working in a school system. The Los Altos principals and school board, they set up some parent meetings, and the parents ñ they were all -- so that they felt comfortable with it. They gave everyone the option to opt out; no one opted out. And now, they're getting a lot of push to go, as you can imagine. We're gonna continue to multiple classrooms. You can imagine there are fifth graders where some of the kids aren't doing the chain rule and their parents are getting a little paranoid. [laughter] That little gap is occurring. I'll ask a--." The modern economy demands an increasingly educated workforce, yet higher education continues to become more and more unaffordable. How is this situation rectified? Can 'college' be disrupted?" So, the simple answer is yes. [laughter] And actually, I think this is something, I mean, I don't know if I should go on the record, on video, but I said this at lunch, so no harm saying it now. Actually, I think Google could play a big role in that. Imagine--and the details have yet to be worked out--but imagine where all of the core academic stuff Khan Academy takes care of. If you wanna learn electric circuits, if you wanna learn big-O notation, if you wanna learn capital asset pricing models, Khan Academy can do those academic things. So, imagine if there was a university in Mountain Viewówe'll put it in downtown Castro Street; it has a nice vibe and all of that-- and you literally, I mean, we could even leverage Khan Academy. We know all of the best students are gonna be using Khan Academy already. I think a lot of them are and so, we don't have a, if you started an institution, you don't have this kind of recognition problem. But what you do is the students, you say, "Look, instead of going to MIT or Stanford or Harvard, why don't you come to this new university--whatever you call it; Silicon Valley, whatever; some name-- and the focus is you spend one year at Google, or six months at Google, six months at Apple, six months working on a project for Kleiner Perkins, six month writing an iPhone app, six months working at Facebook. And you do a series of projects; those projects are essentially your transcript. We have seminars. We'll even set up a little bit of a dorm for you so you have the nice college experience and you can walk into each otherís classrooms, each other's bedrooms in the middle of the night and chat about things and all the rest. And instead of you paying any tuition, we're gonna pay you. Each of those companies, instead of, I mean, all of a sudden, when I graduated from college, all of a sudden all of these--Google didn't exist then--but Google-like companies were like, "Oh, here's a smart guy and from a good school. I wanna hire him." And literally four years ago, I had probably 90 percent of the same skills and I couldn't get a job at a local supermarket. I'm serious. Like, I literally applied to a supermarket and I couldn't get a job. And it was just the process of going to one of these schools that was just like a big signal. So, on one side, actually you would get these really smart 17 or 18-year olds immediately doing useful things and we know from recent companies that started, that there's some very innovative things that 18 or 19 year olds can do, and the company's benefit. But even more important, these people are actually getting tangible skills, but they're still getting the academic component because a lot of that can be done self-paced, it can be done in a seminar format, and frankly, the faculty then, it becomes all of you guys. And the management at Google and Apple and the VCs in the areas and the entrepreneurs in the areas, and I think if you have an institution like that, and all of a sudden, people not only do they not have to pay for school, they can actually get paid to go to school and they finish and they're probably gonna get a better salary and better job placement than pretty much any other university in the world. I think just that example would pretty much disrupt education; disrupt higher education. So, we'll have to work out the details. [laughter] >>audience #8: Hi. So, the mastiff actually looks really great and I'm just really excited to see this kind of thing. My question was how it extends to sciences, where sometimes there really isn't a correct answer-- >>Salman Khan: Yeah. >>audience #8: or, at least if you just write a bunch of word problems, physics word problems aren't really physics. There's something more deep going on there and I was wondering how you would, if you had any thoughts on how that extends. >>Salman Khan: Yes. How do we extend that stuff? Maybe itís not as concrete, multiple-choice, or free answer, right answer all the time. Well, what I think there is a lot that is that, is that kind of builds a scaffold, so you would do the traditional kinematics and Newtonian mechanics and the traditional problem sets, you can do in this type of a form factor. And then it can be all self-paced. I think to get to the next level of real, deep understanding, there's a couple things that we could do. We could more simulation-based learning, where as you go through that knowledge map, at some points these simulations unlock, or maybe they always unlock and you have to figure out some things just by observing the simulations. I think a really interesting thing -- one thing that people -- we talk about providing education, but one thing we don't do is we don't credential people right now; we don't, in an authenticable kind of way. So, maybe oral exams is the right answer. Like, you have badges and right now, you're a novice physicist, or whatever, first level physics person, on the Khan Academy. If you want to get to the middle level or the ninja level or whatever we end up calling it, you have to get five other ninja level people to have a Skype oral exam with you. And only then, if four of the five, or five out of five, say "He's ready to be a physics ninja," that's too cheesy of a name, but then he will move up. There's actually a possibility here to actually do the best of, I mean, the gold standard in evaluation is the oral exam. But it has to be well done. But I think there is something here where we could go to that level. >>audience #8: Thank you. >>Salman Khan: I'll take, "Currently, most of the content is focused on learner-centric approach. Are there plans to incorporate content for instructors/facilitators- more softer skills? Example: Presentation skills, facilitation skills, adult learning theories, etc." We will. I mean, we will do, I don't know if it'll just be for instructors/facilitators, but, I don't know -- this might ruin the Khan Academy, but I've been tempted to give dating advice on the -- [laughter] My son's two years old right now and the theory is, when he's 17, he's more likely to listen to 34 year old Sal than 50 year old Sal. And so, if I just record them now, it'll be like a nice time shift of-- [laughter] I'm kind of serious about -- but-- [laughter] we're open to it. I mean, we don't wanna do something that's really cheesy, like "Make sure to eat five different groups of fruits and vegetables." I mean, some people have lobbied for videos like that. We want people where any Khan Academy video you watch, there's kind of line an "aha" moment, like "Yes. I understand something about the world that I'm not getting from other sources." We would want to do it so it's actually meaningful, but we're not opposed to doing softer skills type of things. We'll see how it works. We don't know. >>audience #9: So, it's clear to me how this stuff can work really well at the elementary level, but I'm more worried about what it might do to the curriculum at the higher levels. I have helped my kids learn this stuff all the way up and by the time you get to Euclidian geometry, a good problem might take half an hour to an hour to solve. The answer might be half a page proof and there might be ten different proofs that are equally valid. I've actually had some specific problems that I've worked through with my son. So how do you cope with the fact that you're limited to this form factor that you described to us? >>Salman Khan: Yeah. I think some of that can be tackled--this form factor--not all of it. I mean, if we designed the modules right you could actually probably do fairly deep, I mean -- it would be hard to be able to have someone actually write the proof and then do a proof check. I mean, even people have attempted-- >>audience #10: [ ] >>Salman Khan: OK, so it's done. Computers can do it. >>audience #9: Not as a practical matter. >>Salman Khan: I mean seriously -- I think we're gonna try to do, we're gonna try to tackle as much as you can do with that form factor, but what we're hoping is if you get a lot of the blocking/tackling done this way, it frees up a lot of time so that when there actually is human interaction, and a human interaction might be with a parent, it might be with a teacher. We have these Meetups happening and these Meetups might turn into spontaneous tutoring sessions, where people can work through hard, more difficult problems, or group problems. I think some of that stuff could, I mean, we're even entertaining, I mean, now we have enough students where we're even entertaining Khan Academy math competitions. It's the next level where it's like human written, unique problems and that maybe teams in different areas can try to solve it. Maybe they can even collaborate. We haven't thought about it deeply yet, but I think it's doable. It won't just be a hundred percent automated, though. >>audience #9: Right. And in a much shorter question that perhaps has the same answer, I coached Math Olympiad for a number of years and one of the things I really enjoyed was when the students came up with a solution that was not the one that I was gonna suggest. You kind of cut the feedback loop here. Is that a problem? >>Salman Khan: Yeah, I think on Math Olympiad type questions, I think that's going to be a completely different-- it won't be this. It will be, so we are playing with the notion of having a question repository, human-written questions. Although, right now, we're thinking it would still be more the right answer type of things, not proof-based. But you could imagine, actually, that would get closer to the peer-to-peer writing type of thing. So, there's a question, people submit solutions and then if you have a critical mass of people on the system, they can evaluate it--kind of a peer review-- and they can say, "Whose proof was the coolest?" or something like that. I don't know. So, it won't be automated in the instruction but because of the Web and the number of people you have, kind of a critical mass, you could probably get a good peer-to-peer thing going, I think. >>audience #9: All right. So, one last try at the first question. >>Salman Khan: Yeah, yeah, yeah. >>audience #9: Because you didn't really answer what scares me most. I'm worried that this will be so successful that it's gonna dumb down the curricula. It's going to be easy enough, maybe it's gonna be so much easier for the school to do stuff in this way, that things that aren't amenable to automation will simply vanish from the curriculum. Is that a legitimate worry? >>Salman Khan: I don't think so. I hope the first part about your worry is true; that it's wildly successful. [laughter] But I think the answer is that right now what you have in schools, you have great teachers and you have not so great teachers and you have things in-between. What happens is, the kids who have the not so great teachers, they're still gonna be able to do well on the SAT. They're still gonna be able to do well on the AP Calculus. They're not gonna be math Olympiads, but they're gonna get the blocking and tackling, so that they still have the same opportunities and all that. And the students who have the good teachers, the great teachers, those teachers are gonna see this as "Yeah, this is taking all the blocking and tackling." Like, if you're tutoring kids, you can say, "Look, you need to learn how to add fractions. I know you can do that on your own, especially if you're gonna get to the Math Olympiad. You can do that part on your own. Khan Academy has your back, but then with the time with me, we're gonna go to the next level." I think, across the board, hopefully it'll have an upward effect on the level of discourse. I mean, it's like a lot of people are gonna have their gaps out of the way, so just everything, hopefully, will be more intelligent in the classroom. Hopefully I'll take -- and anyone should feel free to leave, I'll hang out here and answer a few more questions over the next, I think we're out of time, but, I'll do another ten minutes of this and then we can just whatever. OK? "Do you have any ideas about how to engage parents in their child's development?" The easiest way to do that, any parents, is actually you should log on to Khan Academy right now, and you should start doing the math problems. Although, I suspect, many of you already know a lot of this material. But we've seen over and over again, that that fundamentally changes the conversation that happens at the dinner table; when parents are also doing the same mathematics as their kids. I'll take one. >>audience #11: Is there a chance that kids are just kind of like figuring out the trick and sort of getting through the thing, but there's no real retention? Like ñ do you do anything? >>Salman Khan: No, no. Not only is there a chance, they are doing that. The kids are, the fifth graders especially, it's amazing how good they are in gaming things, at figuring out "Oh, that can't be the choice, that can be the choice." And so, it's a mixed blessing. One is, we are watching them and we're saying, "Oh, they're just gaming that module." So, we either have to make that module less gameable, make it free answer, or do something to it so that it can't be gamed. Or, we have to add other modules to make sure that they're really learning the concept. In terms of retention, we actually do have a thing that after X days, if you haven't done a module or another module that covers that module, intrinsically covers that same concept, it asks you to review it, so you get extra points to go back and review. So, the model is, the ideal school would be, you're walking down the hallway in seventh grade and all of a sudden, your fifth grade math teacher comes up to you and asks you a fifth grade math question. And so, that's what we're trying to do. It is gaming, but it is actually interesting because we actually saw that with the chain rule module. A couple of fifth graders just figured out mechanically how to, even that's not easy though, but they figured out just the pattern. But it was still cool. Well, one, I'd argue a lot of twelfth graders are also doing that, but it was cool once they figured out the pattern. They started to teach each other. So we actually even have this one view, you can see the cascade. Who's the first student in the class and you can see the cascade of the other students right after that, after one person gets it. But, with that said, is we did identify that as -- we have to add more modules to make sure people really are understanding the concept as opposed to just learning the mechanics. >>presenter: So, if you just take one more question and after that, we have to basically wrap up for the video also. But, as he said, he'll be hanging around and you guys can come over and basically chat [ ]. >>Salman Khan: OK. I'll take it right here. >>audience #12: So, as you said, the Khan Academy is definitely turning things around, and how, in the classroom with this kind of self-instructed learning, how do you try to reapply the framework of a grading system or a testing system to give the same kind of progress feedback that people are expecting from education? >>Salman Khan: Yeah, so the question, how do you do the same type of assessment, or how can we start labeling kids if we're--. I think it's a good problem to have. Maybe we shouldn't and I think the ideal is you take -- I actually do think it is good that we do have these standardized tests, the SAT and the AP test, but the ideal is you don't take them when you're 17, like everyone lockstep takes them when you're 17; you take them when you're ready for it. So, there's some fifth graders and they're doing the Khan Academy. When we have the 300 modules and we're like, "Look, statistically, it looks like if you go and take the AP Calculus test right now", we can even give them a distribution. "We think there's an 80 percent chance you're gonna get a five, a four or a five or something. Go take it." And if there's a 17 year old who we think ñ "there's an 80 percent chance that you're not gonna get higher than a three, don't take it yet. Just keep working. Wait until you're 20, wait until you're 25, whatever. If your goal is to learn calculus". One thing I'm hoping we can do, in a non-touchy feely way, cause a lot of people, when people say "no grades," it sounds very touchy feely. It's the opposite. We actually want people to master; everyone should be an A+ student. Everyone should have a hundred percent mastery if they really wanna get to some future level. I mean, I can't tell how many, even when I've interviewed, even in my past life, people who've gone through algebra, trigonometry, geometry, calculus, blah, blah, blah, and they still don't know when to apply algebra. What was the whole point of this non-mastery based learning? It was just a big jumping through hoops and showing that you know how to do homework type of exercise. So, yeah. I think it'll be a good problem that if there's no grades, and if there's literally just, "We say you know this." And if you don't know this at A or B level, if we say you know it, you know it at an A level. Cool. >>audience #11: Thank you. >>Salman Khan: Thank you. [applause]