From Extended Reality to Agentic Reality
Put on a headset today and you are, in a sense, still looking at an elaborate stage set. Objects wait to be picked up. Menus wait to be clicked. Scenes wait to be walked through. Extended Reality (XR) has spent most of its life as a place beautifully rendered, increasingly responsive, but a place nonetheless, where the person is the only one actually doing anything.
Researchers now have a name for what changes when that stops being true: the “agent paradigm.” One recent framework for understanding XR technology separates it from the rest of the stack hardware, virtual-world content, embodiment as its own layer, built from generalist agents, multi-agent planning, and dynamic adaptation to the environment [1]. Agentic AI, in other words, isn’t being proposed as a feature bolted onto XR. It’s being proposed as a fourth pillar of what XR is.
What makes an agent “agentic”
It helps to be precise about what separates an agent from a script:
- A scripted character follows a fixed decision tree. It reacts to input; it doesn’t decide.
- An agent, as studied across XR and AI research, is expected to show some degree of autonomy building trust with people, coordinating with other agents sharing the same space, and modelling behaviour like personality or gait rather than just animating pre-set poses [2].
A large scoping review covering 311 papers at the intersection of XR and AI found that interaction with intelligent virtual agents was one of only five major research directions in the field as prominent as using AI to generate XR content in the first place [2]. Across that body of work, autonomy keeps surfacing as the property researchers actually care about measuring, more than realism of appearance or graphical fidelity [2].
Three places agentic AI is already showing up in XR
1. Agents that cross into the physical world. Most virtual agents are confined to the headset whatever they do stays inside the scene. A research prototype called a “blended agent,” tested in a mixed reality lab study, pushed past that boundary: the agent could act on real physical objects around the user, not just their virtual counterparts, so its behaviour had visible consequences outside the headset too [2]. It’s a small-scale demo, not a shipping product, but the reaction is the interesting part participants called the effect “amazing,” and specifically pointed to seeing the physical consequences of the agent’s actions as what made it feel more present, not less [2]. An agent starts to feel genuinely “there” less because of how it looks, and more because of what it visibly changes.
2. Agents you talk to instead of operate. Generative AI changes the interaction model itself: agents you ask for things in ordinary language, instead of triggering with a fixed input. In prototype AR systems, a vision-language model recognises what’s in a fridge and suggests a recipe unprompted [3]; in mixed reality training, multimodal language models align instructions to a real physical workspace nearly as capably as a human assistant would [3]. A parallel review of generative AI in XR points to the same shift from the content side: systems like AIsop use a language model to autonomously generate an entire narrative experience in VR, deciding not just what things look like but what happens next [4].
3. XR as the place where agents are trained, not just deployed. The relationship also runs the other way. A systematic review of the AI–XR combination found that a large share of the field uses virtual environments precisely because they’re a safe, repeatable place to develop autonomous behaviour: robots and self-driving cars trained entirely inside XR-style simulations through reinforcement learning, then deployed into the real world without further adjustment [5]. Agentic Reality runs both ways agents act inside XR, and XR is where many of today’s agents first learn how to act at all.
The part nobody has figured out yet
None of this is a solved problem, and the research is upfront about that.
Most of what we know about virtual agents comes from asking people how trustworthy or “present” an agent feels not from measuring what it actually gets right [2]. An agent can come across as warm and capable while still quietly getting things wrong.
Then there’s the practical stuff that shows up the moment you try to build one for real: agents that confidently make things up, responses slow enough to break the illusion, and the sheer computing power it takes to run a capable reasoning model in real time [3].
Put those together and you get the real question behind Agentic Reality not “can we build agents that feel convincing,” which is largely solved, but “how much should we actually let them decide,” which isn’t.
A backdrop that decides
Extended Reality asked how far a virtual world could stretch around a person. Agentic Reality asks a different question: what happens once that world can act and decide on its own initiative, both inside the headset and, increasingly, in training the very agents that will later act elsewhere.
It’s a question projects like XR5.0 are already grappling with on the factory floor, not in the abstract: how much initiative a person-centric, AI-powered XR system should take on a worker’s behalf and how to keep that initiative transparent rather than just convenient.
Sources:
[1] B. Zeng, “Recent Advances and Future Directions in Extended Reality (XR): Exploring AI-Powered Spatial Intelligence,” 2025.
[2] T. Hirzle, F. Müller, F. Draxler, M. Schmitz, P. Knierim, and K. Hornbæk, “When XR and AI Meet – A Scoping Review on Extended Reality and Artificial Intelligence,” Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), 2023.
[3] M. Zhu, J. Chen, and B. Li, “When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions,” IEEE Internet Computing, 2026 (preprint arXiv:2601.15308v1).
[4] X. Ning, Y. Zhuo, X. Wang, C.-I. D. Sio, and L.-H. Lee, “When Generative Artificial Intelligence meets Extended Reality: A Systematic Review,” 2025 (preprint arXiv:2511.03282v1).
[5] D. Reiners, M. R. Davahli, W. Karwowski, and C. Cruz-Neira, “The Combination of Artificial Intelligence and Extended Reality: A Systematic Review,” Frontiers in Virtual Reality, vol. 2, article 721933, 2021.
