SENSORAMA

SENSORAMA addresses the urgent need for high-fidelity, adaptive, and ethically trustworthy XR training systems. It introduces a synchronized multimodal data model that integrates visual, auditory, haptic, and thermal rendering with ≤20 ms latency, released through open SDKs/APIs fully interoperable with OpenUSD, OpenXR, and MPEG-I Haptics. Its two-pillar concept combines (i) base technologies – a unified data model with advanced synchronization algorithms, enhanced haptic hardware, and adaptive interaction through neurofeedback (EEG, EMG, eye-tracking) and AI-driven scenario adaptation; and (ii) five high-impact use cases – hip arthroplasty, pedicle screw surgery, dentistry, anatomy education, and industrial assembly – each validated in real-world environments with ≥85% user satisfaction. The project is structured into eight WPs covering requirements, technology development, integration, validation, ethics, dissemination, and management. Ambitious KPIs include: a multimodal glove with >90% realism (M24), neurofeedback-driven interaction improving task performance by ≥20% (M27), five validated demonstrators (M36), ≥15 open-access publications, contributions to at least two standardisation bodies, and zero major ethical incidents with 100% ethics training. The aim of SENSORAMA is to merge visual, haptic (force feedback), tactile, thermal, and auditory cues on the basis of a common, unified data model, and validate its feasibility by way of five high-fidelity training simulators in orthopaedics, neuro surgery, dentistry, anatomy and production technologies. In addition, several haptic and auditory rendering algorithms will be developed; the tactile and haptic devices must be advanced. Further, more variety will be introduced into the training scenarios by leveraging AI, in order to provide a variety of training scenarios, with trainee optimal teaching. Also, the user interaction with the training simulators will include AI speech input, 2D/3D-UIs, and gaze.