What if a training system could understand when an operator is becoming stressed and adapt in real time to provide additional support?
This is one of the key questions being addressed in the XR5.0 project through the development of Human-Centric Digital Twins (HDTs). Rather than treating all workers the same, XR5.0 aims to create systems that continuously monitor human factors and adjust XR experiences according to the operator’s current condition.
The Instituto Piaget team proposed a framework for assessing operators’ stress levels through real-time multimodal assessment. The objective is not simply to collect physiological data, but to combine multiple indicators into a robust representation of the operator’s psychophysiological state.
The framework relies on three complementary sources of information:
- Physiological indicators, such as heart rate and heart rate variability (HRV);
- Behavioural indicators, including eye-tracking metrics;
- Self-report measures provided directly by the operator.
Together, these inputs contribute to the calculation of a Composite Stress Index, a dynamic measure that evolves over the course of task execution. Because the index is derived from several simultaneous indicators, it provides a more reliable assessment than any individual measurement alone. This multimodal approach allows the system to better distinguish between temporary fluctuations and meaningful changes in the operator’s workload and stress level.
The image below shows one component of this broader framework: the acquisition of physiological data via a wearable smartwatch. Heart rate measurements are continuously collected and transmitted to the Clawdite platform, where they can be analysed and used to derive heart rate variability metrics. The dashboard visualises these values and their evolution over time, enabling real-time monitoring of the operator’s state.
However, it is important to note that the XR5.0 stress assessment framework goes far beyond heart rate alone. In parallel with physiological monitoring, XR5.0 integrates lightweight self-assessment mechanisms directly into XR environments. During pilot activities, operators can provide quick evaluations of their perceived stress and cognitive workload using simple in-situ questionnaires embedded within the training experience. These subjective assessments complement the objective data collected from wearable devices and contribute to a more complete understanding of the operator’s condition.
The value of this approach lies in its ability to support adaptation. As the Composite Stress Index changes, the Human-Centric Digital Twin can detect situations where stress levels exceed predefined thresholds. When this occurs, the system may trigger adaptive responses designed to support the operator. Depending on the application, these responses could include modifications to training content, additional guidance, changes in information presentation, or other interventions aimed at reducing cognitive effort and improving performance. This concept reflects one of the central principles of Industry 5.0: technology should adapt to people, rather than forcing people to adapt to technology.
Within XR5.0, the Human-Centric Digital Twin serves as the foundation for this vision. By combining physiological, behavioural, and self-reported information, the digital twin becomes capable of maintaining an up-to-date representation of the worker and supporting personalised XR experiences. The result is a training environment that can respond not only to task requirements, but also to the human being performing the task.
The demonstrations developed so far illustrate the general principle of adaptive human–digital twin interaction. While the smartwatch shown in the image represents the most visible part of the system, it is only one element of a wider real-time stress assessment and adaptation framework currently being developed within XR5.0.
As the project progresses, these capabilities will continue to evolve, enabling XR applications that are increasingly aware of human needs, workload, and well-being. By bringing together wearable sensing, behavioural analytics, subjective assessments, and adaptive XR technologies, XR5.0 is helping shape the next generation of human-centric industrial environments.