In this session, we present Socratic Mind, an AI-powered oral assessment tool designed to enhance student learning by measuring conceptual knowledge, higher-order thinking, and verbal communication skills. Based on pilot study results, we explore how to leverage Generative AI to create a scalable assessment platform to profoundly impact student success.

Cultivating high-level meta-skills–including critical thinking, creativity, and communication–have become crucial to thrive in a rapidly evolving modern society. However, traditional education systems are not always designed to assess and develop such skills. Socratic Mind is an innovative AI-powered oral assessment tool, rooted in Socratic questioning. It is designed to help address the emerging challenge in education: identifying and developing skills students need to think critically, adapt to new challenges, and solve complex problems creatively. By powering oral assessments with generative AI, Socratic Mind not only challenges students to articulate ideas verbally and reflect on their thinking processes, but also makes these assessments scalable and accessible to a wide range of students.

Socratic Mind’s AI components are designed to enhance oral assessments in several key areas. The AI Question Designer generates high-quality assessment questions across various domains, which can be aligned with educational frameworks such as Webb’s Depth of Knowledge. The AI Socratic Questioner conducts assessments by using Socratic-style questioning and simulating the depth and interactivity of a human questioner. The AI Performance Evaluator analyzes student responses against specific rubrics to provide objective and accurate performance evaluations. Finally, the platform's user-friendly web interface allows educators to create, manage, and conduct assessments seamlessly.

Our team incorporates feedback from instructors and students to refine the platform and assessment methodologies continuously. Through surveys and interviews, we gather insights on how the platform can better serve diverse student populations and adapt to the changing needs of educators. This iterative approach ensures that Socratic Mind remains a dynamic and evolving tool for promoting inclusivity and equity in education. Ultimately, by partnering with researchers and educational institutions, Socratic Mind aims to create a robust system that can be deployed across a wide range of learning environments.
Socratic Mind's assessment approach is grounded in rigorous metrics that provide a comprehensive view of student learning. The platform currently focuses on customizable top-down metrics that focus on assessing students’ understanding of core subject matter and content mastery. These metrics allow educators to measure not only students’ basic understanding of course materials but also their higher-order thinking skills, such as the ability to demonstrate reasoning or apply concepts in new and real-world contexts.

Looking ahead, we envision expanding our platform to include bottom-up metrics focused on assessing students' meta-skills. These include the depth of analysis, creativity to counter flawed assumptions, and independent thinking demonstrated in their responses, as well as their ability to engage in meaningful dialogue and construct logical arguments. For example, the AI tool evaluates students’ ability to identify logical fallacies, generate diverse solutions to problems, and formulate original ideas. These metrics provide a nuanced understanding of students’ learning progress and help instructors tailor their teaching to individual needs.

Results of our initial pilot study with approximately 600 students, both undergraduate- and graduate-level, indicated early success of the tool implementation in classroom learning across computer science and business course subjects. Participating students generally perceived the value and helpfulness of using Socratic Mind for deeper learning. According to the participants’ survey responses, approximately two thirds of the total indicated that they preferred the interactive Socratic Mind questions, compared to other conventional quiz or exam questions. Furthermore, more than three quarters of the total reported that they felt they understood the course material better after doing the Socratic Mind part of the assignment. It is also notable that many students found the interactive assignment afforded by Socratic Mind to be more educational than the conventional, short-answer questions.

Based on our ongoing pilot results, this session highlights two exciting features of Socratic Mind: AI-assisted question design and interactive assessment experiences. Creating effective oral assessment questions can be challenging, but our platform streamlines this process through two flexible approaches. First, instructors can simply upload lecture materials or share links to videos and webpages–our AI Question Designer will extract relevant topics for instructor selection and then generate tailored assessment questions. Alternatively, instructors can start with a rough concept and collaboratively refine it through an iterative dialogue with our AI Question Designer until they create the ideal question.
In addition, interactivity is another key feature that sets the Socratic Mind platform apart. Students engage in real-time dialogue with an AI-driven questioner, who uses Socratic questioning techniques to probe their understanding and challenge them to think critically. This interactive process requires students to articulate their thoughts clearly, identify gaps in their understanding, and develop solutions to complex problems. The result is a dynamic learning experience that goes beyond passive content consumption and actively engages students in the learning process.

The topic of our session is particularly relevant to those educators who are interested in exploring novel and AI-driven technologies and tools that can help enhance student engagement and learning. Our session aims to offer opportunities for attendees to gain insights into how generative AI technologies shape the future of education across online, blended, and classroom learning environments.

During the presentation, we will share with attendees several thought-provoking reflection questions about the challenges and opportunities of emerging education technologies for digital learning. The presentation will also include a brief live demonstration of Socratic Mind’s capabilities. Attendees will have an opportunity to interact directly with the AI Socratic questioner, responding to questions designed to stimulate deep thinking and verbal articulation. This hands-on approach ensures that participants not only understand the pedagogical underpinnings of Socratic Mind but also experience its practical applications.
The session will emphasize practical takeaways for educators looking to enhance students’ learning with AI. First, attendees will gain a deeper understanding of the value of oral assessments in promoting student engagement and critical thinking. Research has consistently shown that oral assessments can significantly enhance student understanding, motivation, and confidence across a wide range of disciplines, from STEM fields to the social sciences. Socratic Mind builds on this research by providing a scalable solution that allows educators to implement oral assessments in their own classrooms without being constrained by time or resources.

Second, attendees will learn how AI can be used to create interactive and personalized learning experiences. Socratic Mind’s AI questioner not only conducts assessments but also provides immediate feedback on students' performance, offering insights into their strengths and areas for improvement. This personalized feedback is crucial for helping students develop the meta-skills they need to succeed in a world increasingly influenced by AI. Attendees will leave the session with innovative ideas around integrating AI-driven assessments into educational practices and strategies to intellectually challenge students to think critically and communicate effectively.

Third, the presentation will highlight the importance of assessing students’ learning beyond content knowledge and leveraging the AI assessment tool to develop meta-skills like critical thinking, creativity, and judgment. These skills are essential for preparing students to navigate the complexities of a rapidly changing world. Socratic Mind provides a tool for assessing and nurturing these skills in a way that is both scalable and impactful. Attendees will leave with a clear understanding of how they can design assessments that move beyond rote memorization and foster deeper cognitive development.

In conclusion, Socratic Mind offers a powerful, scalable solution for enhancing student learning through interactive, AI-driven oral assessments. By focusing on the development of critical thinking, creativity, and communication skills, the platform equips students with the meta-skills they need to thrive in a world increasingly influenced by AI. Through personalized feedback, rigorous assessment metrics, and a commitment to inclusivity, Socratic Mind provides educators with a tool for transforming their teaching practices and preparing students for future success.