[TMP-015] AI Aviation Assistant: An Intelligent Pilot Support Tool
The project develops an AI-driven XR aviation assistant to enhance situational awareness in general aviation through 3D visualizations and pilot feedback.
Pilots often need to respond to unexpected in-flight events, requiring quick decisions based on strong situational awareness. While modern technologies like GPS have enhanced decision-making and reduced workload, current cockpit systems still struggle to effectively represent 3D airspace. To address this, we aim to develop an AI aviation assistant that integrates relevant aircraft data, delivers actionable recommendations, and visualizes them in 3D for efficient interpretation.
Extended reality (XR) offers the potential to enhance pilots’ perception with live 3D visualizations of key flight information, such as airspace structure, traffic, and airport highlights. However, XR applications in aviation are primarily used in military and advanced commercial aircraft, leaving general aviation—non-commercial, often single-pilot operations—underserved. This sector could benefit significantly from improved situational awareness and reduced workload.
We plan to create a Unity-based XR application for use with XR glasses. Initial development will involve iterative testing with pilot feedback in a virtual reality flight simulator. Based on these findings, we aim to refine the system and explore the feasibility of a real test flight with our partner, ENAC, using augmented reality headsets like HoloLens 2. Our work focuses on advancing AI-human collaboration, addressing real-world challenges in transportation.
We explored AI and Mixed Reality for pilot support. One of the results includes an early mixed reality prototype for a popular consumer-grade flight simulator that allows to intuitively perceive actual 3D information that current 2D tools cannot present satisfactorily. Based on this mockup, we conducted a very early exploration into AI support strategies that would allow, for example, to convert air traffic control instructions to flight path renderings.
Partners:
- Ludwig-Maximilians-Universität München (LMU), Florian Müller
- Ecole Nationale de l’Aviation Civile (ENAC), Anke Brock