Industrial Extended Reality (XR) systems are often evaluated using familiar measures: Was a task completed faster? Were fewer errors made? Did the system respond reliably? These questions matter, but they capture only part of what Industry 5.0 aims to achieve.
If an AI-enabled XR system helps a technician complete a task more quickly, this represents an efficiency gain. But what happens to the technician’s knowledge and competence over time? Does the system help workers understand their tasks, develop skills and become more confident when dealing with unfamiliar situations? Or can increasingly capable assistance make workers dependent on the technology?
As XR5.0 moves towards its final pilot evaluations, these questions are becoming an important part of the project’s socio-technical evaluation work led by ATB.
Looking beyond technical performance
XR5.0 combines Extended Reality, Artificial Intelligence (AI) and Human Digital Twin technologies across six industrial pilots. Applications range from robot commissioning and maintenance to smart water infrastructure, aircraft maintenance, industrial training and customer support.
The project’s early prototype evaluations already looked beyond whether the technology worked. The common evaluation framework combined technical performance with usability and user experience, work impact and effectiveness, task load, and social and economic aspects. This approach allows XR5.0 to examine technological performance together with the experience of the people expected to use these systems in practice.
This becomes increasingly important as XR5.0 applications combine several forms of intelligent assistance. Workers may receive visual XR guidance, interact with AI assistants through natural language, access recommendations based on technical documentation or receive personalised support adapted to their context.
Pilot 1 provides one example. Its RoboSpace environment combines augmented-reality visualisation of industrial robots with live operational data, the voice-controlled KUKABot assistant, XR training, worker-movement prediction and a Safety-Risk Zone Checker. Evaluating such a system therefore involves more than measuring response times or AI accuracy. It also raises questions about situational awareness, learning and safe human-robot collaboration.
What does the worker learn?
As AI becomes embedded more deeply into industrial workflows, the distinction between using a digital tool and collaborating with an intelligent system becomes less clear.
A conventional digital instruction might simply tell a technician which step comes next. An AI-enabled XR system can potentially explain a recommendation, retrieve relevant technical knowledge, answer questions or adapt information to the user and situation.
Within XR5.0, ATB also leads work on advanced AI paradigms for trusted Human-AI collaboration. The project investigates approaches including active learning, neurosymbolic AI and generative AI. A common principle is that humans should remain active participants in the interaction with AI rather than becoming passive recipients of automated decisions.
This changes an important evaluation question. Instead of asking only “How much faster did the worker complete the task with AI?”, we should also ask “What did the worker learn while using the AI?”
Connecting efficiency with skill development
The first XR5.0 evaluation cycle used qualitative and semi-quantitative indicators to identify strengths, limitations and development priorities across the pilots. For the final evaluation, the project is extending this work towards more precise measurements of efficiency, cost, safety and sustainability while strengthening its human-centred perspective.
One direction identified in this work is an Efficiency-Skill Ratio. The underlying idea is that productivity improvements should not be considered independently from their effects on human competence.
Consider two hypothetical assistance systems. The first substantially reduces task time, but the worker mainly follows instructions without understanding the reasoning behind them. Without the system, completing the procedure becomes difficult. A second system may provide a smaller immediate efficiency gain but helps the worker understand the process, recognise relevant patterns and deal with similar problems independently later.
If execution time were the only criterion, the first system would appear better. From an Industry 5.0 perspective, the assessment is less straightforward.
The Efficiency-Skill Ratio is therefore being explored as an evaluation direction rather than as an established standard metric. It highlights the balance between efficiency achieved through technological assistance and the human capabilities maintained or developed while using it.
Assistance should not always increase
This question becomes particularly relevant for personalised XR systems. A novice encountering a task for the first time may benefit from detailed instructions and explanations. An experienced technician performing the same task may need only occasional assistance.
Effective personalisation may therefore mean learning not only what assistance to provide, but also when assistance should decrease.
Reducing assistance can give workers opportunities to recall knowledge, make decisions and retain competence rather than continuously outsourcing these activities to an AI system. This perspective connects personalised XR with a broader objective of XR5.0: technology should augment human capabilities rather than simply replace them.
Towards the final XR5.0 evaluations
The final evaluation phase will put increasingly mature XR5.0 solutions into realistic industrial settings. The socio-technical evaluation work will build on the experience gained from the early prototypes to refine the common assessment approach across the pilots.
Technical reliability and industrial efficiency remain essential. But the project is also examining a broader measure of success.
Human-centred industrial AI should not be evaluated only by how much work technology can perform for people. We should also consider what people can do with it, learn through it and still do without it.
For XR5.0, this distinction is central to moving from more intelligent automation towards genuinely human-centred Industry 5.0.