[TMP-048] Exploring Multi-Modal Interaction Concepts to Support Users in AI-aided Co- Creation of 3D Objects in Virtual Environments
The project aims to develop efficient and effective ways to create 3D objects using generative AI by exploring various interaction modalities like voice, touch, and gestures in a virtual reality environment.
The use of generative AI in the creation of 3D objects has the potential to greatly reduce the time and effort required for designers and developers, resulting in a more efficient and effective creation of virtual 3D objects. Yet, research still lacks an understanding of suitable interaction modalities and common grounding in this field.
Objective:
The objective of this research project is to explore and compare interaction modalities for collaboratively creating virtual 3D objects with a generative AI. It aims to investigate how input modalities like voice, touch, and gesture recognition can be used to generate and modify 3D objects, and how to develop methods for establishing common ground between the AI and users.
Methodology:
The project consists of two working packages: (1) Investigating and evaluating multi-modal inputs to modify 3D objects in VR. (2) Developing a prototypical multi-modal VR interface for collaborative 3D object creation with a generative AI. This includes AI-generated 3D models or suggestions based on user input. The project will use experimental and observational methods to assess the effectiveness and efficiency of these concepts, through controlled experiments and user behavior analysis.
The project was conducted and finished and a paper submission in a top-tier conference is currently under review. Thus, the authors requested the obscuring of results for now due to the strict anonymization policies of the venue. An abstract version of the results is that the project contributed insights into the effectiveness and efficiency of different modalities and AI assistance in enhancing the collaborative process, and guidelines for the design of multi-modal interfaces and AI assistance for collaborative creation of 3D objects.
Partners:
- Københavns Universitet (UCPH), Teresa Hirzle
- Ludwig-Maximilians-Universität München (LMU), Florian Müller/Julian Rasch
- Saarland University, Martin Schmitz