Lessons Learnt from XR5.0: Five Takeaways on Human-Centric Adaptive Systems

As XR5.0 reaches its conclusion, it is time to look back at what the project taught us. Over its course, we developed and tested AI-based Extended Reality (XR) applications that adapt to the worker, in XR5.0 industrial pilots ranging from robot safety to aircraft maintenance training. This post collects five lessons that will guide our future research and activities.

An accurate model does not make collaboration better

When a robot works next to a person, a natural assumption is that a more accurate prediction of what the worker will do leads to better collaboration. Our experiments tell a more nuanced story. An intention prediction model reached over 98% accuracy, yet collaborative accuracy ranged from 49% to 94% across three scenarios, tracking how predictable the task sequence was. Comparing three cobot logics, intention-based behaviour improved perceived collaboration over random task assignment, but slightly reduced efficiency compared to a deterministic optimal scheduler. What mattered was the interaction built around the model, not the model alone. 

Prediction helps only in the right workspace

In one XR5.0 pilot, carried out with KUKA, a Human Digital Twin (HDT), a live digital representation of the worker, predicts worker movements around mobile robots using multi-camera pose estimation and a lightweight, training-free Kalman-filter tracker. We use only the one-second-ahead prediction for decisions, prioritising deployability over long-horizon accuracy. In our studies (18 participants in the laboratory, 4 operators at the KUKA site), anticipation brought clear gains in smoothness and coordination where worker and robot paths cross, but neutral-to-negative effects in a narrow shared corridor, where non-deterministic speed changes made the robot harder to read. Anticipation is therefore a design choice, not a universal improvement. 

Adapting to each worker has a cost for the team

Adaptive User Interfaces change what a worker sees according to their behaviour and skills. In our experiments with 40 participants, a behaviour-driven protocol reduced assembly errors and performance variability compared to static alternatives, with experts preferring visual feedback and novices text. But the same data suggested a side effect: while each worker became more consistent, differences between workers grew, because every operator interacted with a differently configured system, which can make teams harder to coordinate. 

Learning from an expert does not mean copying the expert

In the aircraft maintenance training pilot, technicians learn valve maintenance procedures with a shadowing module we developed with our partner IML. The movements of an expert technician are recorded through an XR headset and replayed as an overlay, while the trainee’s movements are monitored and compared with the expert’s. Rather than enforcing an exact replication of the expert’s trajectory, the module segments each procedure into operational phases and checks that the critical milestones of each phase are reached. Feedback is therefore robust to different execution speeds, personal motion styles and left- or right-handed execution, and its thresholds adapt to the task phase and the trainee’s skill level. An expert should be a reference for what must be achieved, not a template for how to move.

XR is a means, not a goal

XR headsets promise hands-free, spatial guidance, but they are also a piece of equipment worn by a person. At the KUKA site, the headset delivering safety alerts appeared to increase physical demand, and differences in trust seemed to follow operators’ tolerance of the headset rather than prediction quality. Other XR5.0 pilots, developed by project partners, show the other side: mobile devices were the efficient starting point for remote maintenance of laboratory equipment, and mobile AR suited field operators inspecting water pipes.

The XR5.0 project, funded by the European Union’s research and innovation programme (GA: 101135209) and the Swiss State Secretariat for Education, Research and Innovation (SERI), aims to develop, demonstrate, and validate a novel Person-Centric and AI-based XR paradigm tailored to the requirements and nature of I5.0 applications.

Authors: Davide Matteri, Vincenzo Cutrona, Elias Montini, Samuele Dell’Oca, Sara Masiero, SUPSI, SPS lab