Integrating Large Language Models into Extended Reality Environments for Enhanced User

Recent developments in the fields of Extended Reality (XR) and Large Language Models (LLMs) are transforming the way we interact with technology, how we communicate with it and how it understands us. Integrating these two technologies has the potential to create environments that are highly interactive and dynamic, introducing the concept of Extended Intelligent Reality, a paradigm where immersive XR experiences are enhanced with intelligent, context-aware interactions powered by AI models. In industrial settings, where users frequently have their hands occupied or require immediate and accurate information, such integration can significantly streamline tasks, reduce cognitive load, and improve productivity by providing real-time, precise, context-aware support. This paper presents a prototype, “LlymX”, developed to demonstrate that this integration is feasible. The system is built around a 3D immersive environment developed in Unity, backed by a LLM-driven Python backend leveraging embedding algorithms and retrieval-augmented generation (RAG) techniques via vector databases such as Chroma DB, managed by the Lang- Graph framework. LlymX allows natural, intuitive interaction through multimodal input methods like speech, gestures, and text, enabling users to easily identify objects within a 3D scene and retrieve detailed, contextually accurate information. Future developments of LlymX will involve user-centered validation studies in realistic industrial scenarios to quantify benefits such as reduction in task completion time, improvements in error rates, and enhanced user satisfaction. Additionally, expansion into other application domains, including healthcare and professional training, represents a promising avenue for future research.