Khan Academy annual report 2023-2024 - Districts Partnerships program

Khanmigo, Khan Academy’s AI tutor and teaching assistant, was originally piloted in March 2023 with two district partners serving historically under-resourced communities.

This was a critical aspect of our pilot program’s design: to improve student learning outcomes across the board, we must design educational tools that work for everyone.
‍
Students in our pilot districts are, on average, more likely to be eligible for free or reduced-price lunch and tend to be more ethnically diverse than those in U.S. benchmarks. Piloting Khanmigo in these types of districts was an intentional product-design decision. We know that systemic factors have created a tremendously unlevel playing field for under-resourced students. We believe it’s our responsibility to bring Khanmigo into classrooms in a way that empowers all students rather than creates a deeper digital divide.

Designing and refining Khanmigo with educators in the field is also an intentional product design choice. From the time we began sharing Khanmigo through our pilot program, we have partnered with educators to truly understand how we need to adapt and evolve Khanmigo to meet the needs of the classroom. Educators are part of a continuous feedback loop that starts before Khanmigo is implemented. Because each Khanmigo implementation is unique, educators in each district, from one classroom to the next, provide unique input on what Khanmigo needs to do to help students learn. Following implementation, we meet with educators regularly to share knowledge and hear feedback on what’s working for them and what we can do better. Feedback we receive in these meetings is shared with our product and user experience teams, who make iterative adjustments and occasionally conduct additional in-person research in our district partners’ classrooms to raise the bar on the student learning experience.

During the 2023-24 school year, Palm Beach School District implemented Khanmigo alongside KAD (Khan Academy Districts) to focus on three areas: intensive reading, digital SAT preparation, and Advanced International Certificate of Education (AICE) / Advanced Placement (AP) programs. Khan Academy collaborated with Palm Beach educators to help achieve the following:

Developed custom implementation resources designed to address Palm Beach’s focus areas
Co-developed daily lesson plans aligned to Palm Beach’s scope and sequence that incorporated KAD content and ideas for utilizing Khanmigo in each lesson
Provided live-virtual training to supplement ongoing professional learning sessions. Teachers told us they needed additional support with Khanmigo. We provided >60 live-virtual learning sessions for educators to help them feel comfortable with and make the most of Khanmigo in their classrooms.
Co-created Google classroom resources for Palm Beach educators to access on-demand educational content for teachers
Collected ongoing feedback from Palm Beach administrators and educators to understand how KAD and Khanmigo could be improved to meet their needs. These insights are routinely shared with Khan Academy’s product and user experience (UX) teams to drive product improvements.

Throughout the 2023-24 school year, 53 districts partnered with us to bring Khanmigo to classrooms across the U.S., reaching ~68.2K students.

In addition to collecting pre-implementation input and feedback via professional learning sessions, we’ve also collected feedback through a combination of in-person user-research studies, focus groups, and in-product feedback. We are always working to improve the Khanmigo user experience, using input from our users. While there will always be more to learn to ensure Khanmigo meets learners’ and educators’ needs, here are our recent lessons learned as well as some information about what we’ve done to put our learnings into action:

Educators and learners need help understanding how to get the most out of AI: AI is a novel tool for all of us. We had anticipated that educators and administrators would need support and ongoing learning to integrate Khanmigo into their schools and instructional programs. We ramped up staffing and developed new professional learning modules to support educators as they experiment with using AI in classrooms. Even with this expansion, we heard requests for even more learning and support from school district leaders and teachers. We met this need by providing additional support and professional learning sessions personalized to individual districts, and in some cases, individual schools. Over the past year, we conducted a total of 581 learning sessions with our district partners to ensure they were comfortable with and equipped to utilize Khanmigo in their classrooms. Looking ahead, we are anticipating the need for support to remain steady and have hired additional staff to support this need. We are also looking at ways to efficiently scale our professional learning and support models without sacrificing quality or personalization.

Students often interact with Khanmigo in very short sentences, like “idk [I don’t know].” Khanmigo is poised to help the most when it has information about how students are thinking. Responses like these do not give Khanmigo information about what a student knows or does not know. To address this and meet students where they are, we use best practices from the field to adjust how Khanmigo interacts. For example, we use dynamic action bubbles—think of pre-populated question options—that give students suggested prompts to use with Khanmigo. This way, rather than typing “I don’t know,” students have scaffolding to help them ask for help, and Khanmigo has insight into students’ thought processes that it can then use to personalize tutoring. More work is needed to help Khanmigo interact with students effectively and help students make the most of this powerful new learning tool. We are continuing to iterate with our school partners to improve Khanmigo’s tutoring experience, which we will continue testing and assessing.

Khanmigo has room to improve in math computation and math tutoring: Khanmigo makes periodic math mistakes. But getting the math right is just one part of the challenge. The other part of the challenge is making sure Khanmigo evaluates student work correctly. Khanmigo sometimes makes mistakes evaluating whether a student is right or wrong, even when it calculates the math correctly.

With these kinds of challenges, why turn to AI to support student learning in math in the first place? With continued work, these challenges are addressable. More importantly, students need extra support in math.* Though there is more work to do, we are steadily working toward giving students additional math learning support by improving Khanmigo’s math computation and tutoring capabilities. In truth, this is a complex problem facing the entire field, and the starting point for Khanmigo’s accuracy is the accuracy of the technology upon which it’s built. To improve that accuracy, our team of engineers, researchers, and former teachers have undertaken the following actions to improve Khanmigo’s math computation and tutoring:

We’ve built a calculator for Khanmigo to solve numerical problems instead of relying on AI’s predictive capabilities.
We’re assessing various large language models (LLMs) to find the best fit for application in math tutoring.
We’ve re-engineered Khanmigo’s responses so that Khanmigo more closely mimics how a live tutor works with a student.

This last initiative, re-engineering Khanmigo’s responses to mimic a live tutor, is a good example of the complexity that accompanies building an AI-based tutor. On its own, AI doesn’t automatically incorporate research-based tutoring best practices like providing adaptive instruction and feedback or managing students’ cognitive load; we need to prompt it to do that. In addition, students need help interacting with AI. To improve the overall tutoring experience, we need to help students interact with AI and we need to instruct Khanmigo to be a better tutor. To improve Khanmigo’s tutoring abilities specifically, we refined the instructions we send to the large language model (LLM) to guide its tutoring approach (called prompt engineering), which makes Khanmigo more effective at understanding and responding to learner needs. For learners, we introduced interactive features, such as dynamic action bubbles, that give learners options for questions to ask Khanmigo. Khanmigo also suggests which types of questions yield the best responses. Together, these learner- and tutor-experience modifications make Khanmigo's interactions more closely resemble the research-backed practices of successful, intelligent tutors. Over time, as new learnings are gleaned from the field, we will continue to iterate and improve on both the tutoring and the learner experiences.

We are grateful to our district partners whose collaboration over the past year has enabled us to test, iterate, and learn while bringing new, transformative technology into classrooms. Artificial intelligence is still a novel technology. AI researchers, educators, and organizations like Khan Academy are all still learning how we can make the most of this powerful technology in an educational context. It takes all of us working together to ensure the technology is meeting the needs of users, which means providing ongoing transparency about issues and collaborative, continuous work to improve over time.

*Students need extra support in math: According to The Nation’s Report Card, 98% of 9-year-old students in the U.S. are at the most basic proficiency level for their age group, grasping only “Simple Arithmetic Facts.” This means they know some basic addition and subtraction, and most can add two-digit numbers. Similarly, the majority of 13-year-old students (96%) are at the most basic proficiency level for their age group, “Beginning Skills and Understanding.” This means that in addition to basic addition and subtraction, they know some basic multiplication and division.

Driving student learning in historically under-resourced and underrepresented communities

We design with educators in the field to ensure our educational tools work for everyone.

Our district partners have provided valuable feedback that we’re activating to improve Khanmigo’s effectiveness.