About the project
Objective
Large Multimodal Models (LμMs) have the potential to transform engineering education by supporting hands-on, experiential learning. LμMs can process images, audio, video, and other data types, making them ideal for supporting physical engineering design tasks. However, these tools must be carefully designed to align with educational theories and support, rather than hinder, student learning. This project aims to develop and evaluate a pedagogically-aligned virtual teaching assistant (μTA) powered by LμMs to support problem-solving with physical systems in real-world settings for engineering education. The project addresses the challenges students face when dealing with complex, ill-defined problems in engineering design courses and other experiential learning contexts and the limitations of current AI tools in these settings.
Background
Generative AI tools, like large language models (LLMs), have revolutionized education but remain largely confined to screen-based, text-centric tasks such as programming and writing. Recent advancements in Large Multimodal Models (LµMs) enable processing of diverse inputs, such as text, images, and videos, offering opportunities to extend AI’s benefits to experiential learning environments like workshops and labs. While current research focuses on screen-based applications, little is known about how LµMs can support hands-on, ill-defined problem-solving tasks central to engineering education. This project pioneers the integration of LµMs into these settings, co-designing tools with students and educators to foster skills critical for engineering innovation, their studies, and work success.
Crossdisciplinary collaboration
The project is led by two principal investigators from the KTH Royal Institute of Technology: Associate Professor Olga Viberg (Human Centered Technology/EECS) and Assistant Professor Richard Lee Davis (Learning in Engineering Sciences/ITM). This cross-disciplinary collaboration integrates Viberg’s expertise in the design and evaluation of educational technologies—with a strong focus on AI adoption in STEM education and participatory design methods—with Davis’s experience in designing AI-driven tools for experiential learning, integrating multimodal systems, and advancing pedagogical alignment for generative AI technologies.
