Immersive Data Visualisation and Interaction for Teaching Scientific Machine Learning

The project «Immersive Data Visualization and Interaction in Teaching for Scientific Machine Learning» aimed to enhance the field of Scientific Machine Learning (SciML) by introducing advanced data visualization techniques and interactive Extended Reality (XR) applications for teaching purposes. The project recognized the need for effective data analysis and visualization in the context of the increasing volume of data from various sources in architecture and engineering. The motivation behind the project was twofold: addressing limitations in the existing SciML course and preparing students for the future challenges of utilizing AI tools in research and industry.
The existing SciML course provided a solid foundation in AI, Machine Learning (ML), and Deep Learning (DL) for students of architecture and civil engineering. However, an evaluation of the course identified weaknesses in data visualization and interpretation, particularly regarding AI models and their results. The project aimed to bridge these gaps by introducing expressive and decision-supporting visualizations.
The project recognized the potential of interaktive and immersive technologies to revolutionize data analysis, presentation, and communication. By leveraging interactive graphics and XR applications, students were exposed to a new approach to mining, presenting, and communicating data throughout the entire Knowledge Discovery Process (KDP) in SciML. The goal was to enhance students› understanding of data, ML/DL models, and their results by creating an interactive link between different steps of the KDP through immersive data visualization and communication. Another important motivation was to equip students with the skills and literacy needed for future advancements in AI and data science. By using immersive and interactive technologies, the project aimed to foster curiosity, advanced analytical thinking, and abstraction skills necessary for handling complex real-life data problems. Integrating XR into teaching also aimed to modernize and digitize architecture and civil engineering education at ETH Zurich, contributing to the overall development of the curriculum. The tool set was developed agnostic to allow impact and applicability beyond the SciML course and the AEC domain.
Overall, the project aimed to bridge the gap in data visualization and analysis techniques, prepare students for the challenges of utilizing AI tools, and provide an immersive learning experience that aligns with the demands of Industry 4.0 and beyond. By incorporating XR applications into the teaching process, the project aimed to empower students with the skills, knowledge, and curiosity necessary to thrive in the evolving landscape of AI and data science.

As part of the project, we made several modifications to the existing SciML course syllabus. We relocated some background knowledge on Machine and Deep Learning to an appendix document, allowing us to rework the existing data visualisation lecture and to create a whole new lecture on that topic. The first lecture, «Data Processing and Visualization,» was expanded to include an exercise that walks students through the various steps of the knowledge discovery process. The exercise provides Python code in the form of Jupyter notebooks, covering preprocessing techniques and visualizations for different data types. The second lecture, «Understanding, Exploring, and Employing High Dimensional Data in SciML,» focuses on relevant visualization and interpretation topics. It is divided into four parts: dimensionality reduction, immersion using XR, interactive data visualization, and hands-on demonstrations. The dimensionality reduction section introduces techniques such as PCA, TSNE, UMAP, and SOM for visualizing training data and AI algorithm results. The immersion section explores data representation using XR, while the interactive visualization part covers annotation methods. The hands-on session showcases real-world design and analysis problems in architecture and civil engineering, demonstrating how different visualization techniques can be applied. After the sessions, we collected oral feedback from the student audience towards their perception of usefulness, methodical depth and transferability of the content. The overall response was very positive and motivated some student projects in form of semester projects or master theses.

At this point, we would like to give more concrete insight into the developed demonstrators:
Firstly, we explored Neural Network-based sensitivities for examining the impact of input changes on a gridshell’s embodied carbon. Using a conditional variational autoencoder and Gaussian Mixture Models, we extracted insights from data distributions. Students investigated design influences through local sensitivities and interactive interfaces, contextualizing local insights in the broader design space.
Secondly, we developed a data annotation method using VR and WebXR, allowing real-time annotation on any VR/AR device. Our focus was on qualitatively assessing sound diffusion panel aesthetics and retraining an ML model. Immersive annotation was crucial for accurate judgments. Student feedback was positive, inspiring further exploration in this area.
Thirdly, we introduced an interactive data annotation method using Plotly and Dash, offering an alternative to XR technology. Students explored high-dimensional data visualization challenges and design optimization. Their feedback was positive, highlighting the applicability of this method as a pre-study for immersive annotation.

The project successfully achieved all of its goals and even exceeded expectations as publications and further projects departed from this. The developed interactive data visualization and interaction tools proved to be valuable assets not only for the lectures but also for researchers in various fields. By introducing new lectures and exercises, students were equipped with the necessary knowledge and code to apply and further develop these ideas in their own research and projects. The impact on student learning was investigated through various means, including assessments, feedback, and project outcomes. The positive results were evident in the increased engagement and understanding of students. The lessons learned from this project can be translated to other contexts and larger student populations by sharing the developed software tools and teaching materials with ETH colleagues and beyond. By disseminating these resources in the upcoming semester through the SciML lecture but also our presentations on department level and also international conferences, other educators and researchers can adopt and adapt them to enhance their own teaching and research endeavors. The project’s success serves as evidence of the broader applicability and potential impact of the developed tools and methods in advancing data visualization and interaction in educational settings.

Related Publications stemming from parts of this project:
Kraus, Michael, et al. «Improved Perception of AEC Construction Details via Immersive Teaching in Virtual Reality.» arXiv preprint arXiv:2209.10617 (2022).
Kraus, Michael A., and Romana Rust. «Immersive Teaching of AEC Construction Details using Virtual Reality.» Proceedings of 33. Forum Bauinformatik. 2022.
D. Fang, S. V. Kuhn, M. A. Kraus, W. Kaufmann, and C. Mueller, “Quantifying the influence of continuous and discrete design decisions using sensitivities,” in Advances in Architectural Geometry 2023, Stuttgart, 2023.

This project inspired another Innovedum project of Dr. Kraus together with Prof. Kaufmann towards using Augmented and Virtual Reality to communicate model abstraction, structural analysis and layouting of reinforced concrete structures, starting in 2023. Finally, the outcomes of this project will be included as software functions in an AI-based design toolbox from Design++, providing interactive and immersive methods for inspection and interpretation of high-dimensional and mixed data for AEC projects and beyond.

Jose Guanter-Fonds, Visualisation and Simulation