Evaluating the impact of AI-based tutoring systems on student learning in biology education

Acquiring a sound understanding of concepts is a prerequisite for successfully studying at ETH and enables students to tackle interdisciplinary issues and future challenges. Critical examination of one’s own knowledge plays a key role in this. However, the constantly increasing number of students poses a major challenge for lecturers to ensure that the requirements for a personalized education are met in the future. The rapid development and spread of new technologies such as ChatGPT are opening new possibilities for learning support; at the same time, however, it is also clear that their direct use at universities is accompanied by limitations in terms of factual accuracy, didactic embedding and benefits for learning and teaching.

In this project, we developed a lecture-specific AI-based chatbot and investigated its effectiveness on student learning. The open source chatbot «bioTutor» enables a Socratic dialogues grounded in a constructivist learning environment and is specifically tailored to lecture content. This enables an individualized learning experience with personalized feedback and may promote learning by working on biological topics while incorporating prior knowledge. The bioTutor differs from classic chatbots in that, as part of the conversations, further topic-relevant questions are asked, which are answered by the students and thus critically reflected upon. The learning process can be tracked by lecturers via detailed usage analyses in a specifically developed lecturer dashboard so that comprehension difficulties can be identified early on and students can be given targeted support.

The evaluation of student usage and feedback suggested high usability and perceived usefulness for lecture and exam preparation, further supported by the over 10’000 recorded interactions per semester. Thus, the bioTutor demonstrates a scalable and adaptable model for pedagogically grounded AI in higher education.

This project aimed to integrate a lecture-specific and AI-supported chatbot «bioTutor» into biology lectures to optimally support students in developing complex interdisciplinary content through individual learning experiences.
The following goals were pursued.

Goal 1: Integrate and use bioTutor in specific lectures to a) track and analyze student learning and b) increase student engagement and learning performance through continuous development of the tool based on surveys and measurements.

Goal 2: Plan and conduct research and surveys to find out how and for what purpose AI-based chatbots can be used most effectively to best support students.

Goal 3: Establish workflows that allow instructors to analyze usage data anonymously to uncover comprehension issues and track student learning

Goal 4: Develop a paper to support lecturers and teachers in the planning, development, and implementation of a similar lecture-based chatbot or the use of the developed chatbot, investigate the transferability of the results to other contexts, topics and disciplines, and scientifically publish the collected data and findings.

The project shows possible applications of new technologies in teaching and has a potential impact on students› learning process, the design process of teaching materials, and curriculum development.

Added value for students: The availability of a knowledge-based and lecture-specific chatbot for learning new concepts and knowledge review represents a new opportunity for self-study. Specifically, the chatbot is made available to students as a web application for personalized preparation and follow-up of lectures, as well as for support in exam preparation. By constructively embedding the already developed chatbot in various lectures, the academic learning success of the students can ideally be better promoted. In addition, the tool offers the opportunity to acquire AI-specific skills in relation to the use and risks of such tools.

Added value for lecturers: By having students use the course-specific chatbot, lecturers gain insight into the data collected, which provides an overview of prior knowledge, existing misconceptions, and the learning process. This allows lectures and exercises to be adapted accordingly. In addition, the chatbot allows better control over the handling of misinformation from other chatbots, as mainly lecture-relevant content would be available.

Added value for degree programs: In addition to the students directly involved in the project, other students could benefit, as a successful project completion could contribute to changes in individual aspects of teaching. Better knowledge of the level of knowledge of a student cohort is valuable for curriculum development in order to adapt learning content or learning units or to provide additional learning opportunities to close knowledge gaps. Knowledge about the integration and application of such chatbots can also be used in course initiatives to offer similar courses across disciplines and semesters.

With growing student numbers and increasingly diverse educational backgrounds, providing individualized learning support in higher education has become increasingly challenging. Lecturers often have limited capacity to address students’ individual questions and learning needs, particularly in large lecture-based courses. At the same time, recent advances in large language model–based chatbots have created new opportunities to support students’ learning processes outside the classroom.

However, commercially available chatbots often lack accuracy, transparency, and pedagogical grounding when applied to specific university courses. Moreover, their responses are typically not aligned with course materials or learning objectives, which limits their usefulness for structured exam preparation and lecture revision. This gap motivated the development of a didactically motivated lecture-specific chatbot that could reliably support students.

To address this challenge, we developed a chatbot based on lecture materials that supports knowledge acquisition through dialogue-based problem solving. By interacting with the chatbot, students can revise lecture content, practice applying concepts, and receive guidance during their learning process. In addition, the chatbot «bioTutor» provides lecturers with anonymized insights into students’ learning progress through a specifically developed lecturer dashboard, enabling instructors to better understand common difficulties and adapt their teaching accordingly.

A major highlight of this project was the successful development and implementation of an open-source, lecture-specific AI chatbot. What made this implementation particularly successful was the strong acceptance among students. This was reflected not only in the intensive use of the chatbot throughout the semester and during the exam preparation phase, but also in students’ ratings regarding usability and perceived usefulness.Having been able to provide a tool that directly benefited students through personalized feedback while also generating valuable and actionable insights for lecturers through a specifically developed dashboard was another key highlight of the project.

Moreover, the chatbot was presented at various occasions both within and beyond the context of ETH Zurich. For instance, we presented the concept of the chatbot during the «Refresh Teaching» session on generative AI and education (September 24, 2024), at the «AI in Teaching» forum organized by ETH Zurich and UZH (August 28, 2025), and at the «Interdepartmental AI Science Event» at ETH Zurich (November 25, 2025), Furthermore, our project was featured on the ETH «Learning and Teaching Blog» (May 23, 2025), and we presented bioTutor at the annual conference of the Gesellschaft für Medien in der Wissenschaft (GMW) on «Agility and AI in Higher Education» (October 24, 2024).

In end-of-semester surveys and tool-integrated questionnaires, we assessed the bioTutor’s perceived usefulness and ease of use as well as the students’ evaluations of their interactions with the chatbot. The latter included the quality of the conversations for resolving open questions and the usefulness of follow-up questions for deepening their understanding.

The results from these questionnaires showed that the chatbot received high usability ratings and that students generally appreciated its didactic design. Moreover, students indicated that the chatbot’s answers and follow-up questions supported their learning process. These findings are reflected in the tool’s usage analysis, which shows persistently high interaction volumes prior to assessment periods as well as during regular semester phases and even after the course had finished.

Providing a lecture-specific AI chatbot requires the preparation of high-quality lecture materials in an LLM-readable format. While this step is relatively simple when lecture-accompanying scripts have already been developed, for many lectures only lecture recordings and slide sets are available. To convert this information into a text-only format, since language models operate on text, the videos need to be transcribed and manually cleaned to remove disfluencies. In addition, complex images need to be manually described so that the content can be understood without further context. These two steps are resource- and time-intensive.

A further challenge arises from self-developing a chatbot and relying on external functions and libraries. As these components are continuously developed and adjusted, the application itself must also be regularly maintained and updated. In addition, external API limitations, such as request quotas or delays during peak times, can lead to increased waiting times. Integrating the chatbot into the ETH learning technology portfolio would likely increase its usage and facilitate its implementation in other courses at lower cost. Although thisintegration was not feasible within the scope of this project, we are in contact with the ID team on implementing features of the bioTutor within the planned ETH AI platform.

Lastly, clear communication regarding the chatbot’s functionalities and data handling is important. Informing students about the didactic purpose of the chatbot allows them to approach it as intended and limits usage to students who are willing to interact with a chatbot that does not provide direct answers, thus avoiding frustration caused by the chatbot’s unwillingness to provide answers directly. Furthermore, informing students which data is recorded and how it is stored and processed is also important from an ethical standpoint. Finally, collecting student feedback allows the functionalities of the chatbot to be adjusted for its continuous further development.

One of the main goals of the project was not only to provide a didactic framework for AI-based chatbots in higher education, but also to inform and support lecturers in the development and implementation of their own lecture-specific chatbot. To achieve this, we opted for a fully open-source development of the chatbot and dashboard, as well as a modular code architecture that enables educators to easily reuse and adapt the software to specific use cases. By replacing the knowledge base and system prompts, if necessary, a similar system can be adapted for other courses. Since the framework is scalable and flexible in terms of hosting, it can be applied across various disciplines with diverse cohort sizes. An ETH Zurich internal Moodle course is currently developed and will be available for interested lecturers and educational developers. Furthermore, an empirical research paper and documentation will support adoption by other instructors.

Project Website

Refresh Teaching Video

Learning and Teaching Blog

Project Video