Beyond the Virtual Horizon: Safe Navigation of Extended Realities

Throughout the rich evolution of human history, a continuous fascination with reality vs. imagination has persisted. Attempts to capture, recreate, and augment reality have shaped humanity’s understanding of the world. Amid the digital revolution, the quest for alternative realities has taken on a new dimension with the advent of extended reality (XR). This seemingly magical window allows users to see and experience things that are not actually there.

The realm of XR, with its multifaceted landscapes and complex interplay of rewards, risk, ethics, privacy, security, and governance, demands attention. Organizations must understand the transformative potential of XR technologies and critically examine the myriad challenges, including ethical, privacy, and security considerations, inherent in the adoption and proliferation of XR.

Introduction to Extended Reality

XR is a spectrum of technologies that blend the physical and digital worlds to enhance user experiences. It is an umbrella term encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR). Initially driven by the development of VR in the 1960s,¹ XR has since incorporated AR and MR advances. The introduction of affordable, powerful headsets, along with advancements in software and computing power, has propelled XR from niche applications to mainstream adoption in healthcare, education, entertainment, and beyond.

A classic XR environment involves:

• Hardware—Headsets, sensors, cameras, haptic devices
• Software—Applications, platforms, development tools
• Infrastructure—Network requirements, cloud computing, edge computing

An overview of these components interacting with each other is depicted in figure 1.

Source: Meccawy, M.; “Creating an Immersive XR Learning Experience: A Roadmap for Educators,” Electronics, 30 October 2022

XR comprises several categories:

• VR—A fully immersive experience that isolates users from the real world and places them in a completely digital environment. Key features include 360-degree views and interactive environments using VR headsets (e.g., Oculus Rift, HTC Vive).
• AR—An experience that overlays digital content onto the real world, enhancing the user’s perception of reality. Key features include real-time interactions and integration with physical surroundings using AR-capable devices (e.g., smartphones, AR glasses such as Microsoft HoloLens).
• MR—Combines elements of both VR and AR, allowing digital and physical objects to coexist and interact in real time. Key features include seamless interaction between digital and real-world elements and advanced spatial awareness using MR devices (e.g., Microsoft HoloLens 2).
• Metaverse—A virtual world where users interact in a 3D environment to make social and economic connections. Key features include persistent, shared, and interconnected digital environments that often incorporate user-generated content and social interactions.

Importance and Relevance of XR in Various Industries

XR is revolutionizing multiple industries by enhancing user experiences, improving training and education, and streamlining complex processes. Some of the current and potential applications of XR in various industries are illustrated in figure 2.

Rewards of XR

It is apparent that XR technologies offer significant benefits, including:

• Enhanced user experiences—XR creates immersive environments that enrich user experiences, leading to greater satisfaction and engagement in various sectors (e.g., gaming, entertainment, retail).
• Improved training and education—XR offers innovative training and educational methods through realistic simulations and interactive learning, enhancing information retention and practical skills, especially in medicine, aviation, and technical training.
• Enhanced collaboration and communication—XR enables better collaboration and communication by allowing individuals to interact in shared virtual spaces, improving remote work and productivity.
• Opportunities for innovation and creativity—XR opens up new avenues for creativity and innovation, enabling developers to create unique applications and experiences, and driving new products and services across various industries.
• Economic benefits and market potential—XR presents significant economic benefits including job creation, new business opportunities, and substantial market growth, and provides enterprises with competitive advantages and opportunities to tap into emerging markets.

Extended Reality Risk and Challenges

While XR technologies offer numerous benefits, it is critical to address risk and challenges. The challenges span health and psychological impacts, technical issues, accessibility, legal and regulatory concerns, and economic barriers. Understanding these sources of risk is crucial for the responsible and sustainable development of XR:

• Health risk—Extended use of XR can lead to health issues such as motion sickness, physical discomfort, and eye strain. Ensuring proper guidelines and ergonomic designs can mitigate risk and enhance user wellbeing.
• Psychological impacts—XR can cause psychological effects, including addiction and dissociation. Balancing immersive experiences with real-world interactions is essential to prevent negative mental health outcomes.
• Technical challenges—Technical limitations such as hardware constraints and latency issues can affect XR performance and user experiences. Ongoing advancements in technology are needed to overcome these barriers.
• Accessibility issues—XR may exclude certain demographics, such as individuals with disabilities or those without access to necessary technology. Inclusive design and broader accessibility initiatives are critical for widespread adoption.
• Legal and regulatory challenges—XR raises legal and regulatory issues, including intellectual property rights and liability concerns. Establishing clear legal frameworks and guidelines is vital to protect users and developers.
• Economic barriers to entry—High XR hardware and software costs can be an economic barrier for many users and developers. Reducing costs through innovation and economies of scale can help make XR accessible to a broader audience.

Privacy Challenges in Extended Reality

It is clear that XR technologies bring significant advancements but also pose substantial privacy challenges. Addressing these privacy concerns is crucial for building user trust and ensuring the ethical use of XR technologies:

• Data privacy risk—XR devices often collect sensitive biometric data, such as eye movements, finger tracking, and facial expressions. Protecting this data from misuse and ensuring that it is handled with the utmost care is essential to maintain user privacy.
• Location tracking and surveillance—Many XR applications rely on precise location tracking, raising concerns about surveillance and the potential for misuse of location data. Implementing strict controls and transparency measures can help mitigate these risk factors.
• Third-party data sharing and monetization—Data collected by XR devices can be shared with third parties for monetization purposes. Clear policies and user consent mechanisms are necessary to ensure that users know how their data is used and shared.
• User profiling and targeted advertising—XR technologies can create detailed user profiles based on their interactions, leading to highly targeted advertising. Ensuring that user profiling respects privacy boundaries and includes opt-out options is crucial.
• Risk of data breaches and unauthorized access—Like other digital technologies, XR systems are vulnerable to data breaches and unauthorized access. Robust security measures, including encryption and access controls, are vital to protect sensitive user data from cyberthreats.

Ethical Considerations in Extended Reality

The development and deployment of XR technologies come with significant ethical considerations. Addressing these ethical challenges is essential for responsible innovation and user trust:

• Privacy concerns—XR technologies collect vast amounts of data, raising concerns about user privacy and surveillance. Collection of this data can expose sensitive user information, enabling the creation of detailed user profiles. Such profiles could be used to make automated decisions affecting sectors such as the workplace, education, crisis management, and healthcare. Ensuring that data is collected, stored, and used responsibly is critical to protecting users’ rights and maintaining their trust.
• Consent and autonomy—Obtaining informed consent from users is paramount. Users should know what data is being collected and how it will be used. They should also understand how immersive experiences may impact their autonomy and wellbeing.
• Credibility of system responses—To enhance interactivity, personalization, and automation of XR system responses, generative artificial intelligence (AI) models are employed to create dynamic, responsive environments and tailored experiences based on the signals received. However, issues arise with the explainability of these results, as AI’s decision-making processes are often opaque. Transparency is required in such cases, because users should understand AI’s role in their experiences and the accuracy of the results generated by the system. This is vital in fields such as healthcare, where precise and reliable outcomes are essential. Hence, combining XR with generative AI will challenge the robustness of the legal framework, especially given the current perceived lack of regulatory guidelines to address privacy, security, accountability, copyright violations, disinformation, misinformation, and the misuse of generative AI.²
• Content moderation and censorship—Moderating content in XR environments poses unique challenges due to the immersive and interactive nature of these experiences. Balancing the need to prevent harmful content with the principles of free expression requires careful consideration and robust moderation policies.
• Representation and diversity in XR content—Ensuring diverse representation in XR content is crucial for creating inclusive experiences. XR developers must strive to reflect a wide range of perspectives and identities, avoiding stereotypes and ensuring that all users feel seen and valued. Representation and diversity in XR content—Ensuring diverse representation in XR content is crucial for creating inclusive experiences. XR developers must strive to reflect a wide range of perspectives and identities, avoiding stereotypes and ensuring that all users feel seen and valued.
• Ethical design and development practices—Adopting ethical design and development practices involves considering the long-term impacts of XR technologies. This includes prioritizing user wellbeing, ensuring accessibility, and addressing potential societal implications. Ethical guidelines should be provided that can help guide developers in making responsible decisions.

Security Issues in Extended Reality

As XR technologies become more integrated into daily life and across industries, addressing security issues is crucial to protect users and their data.

Malware and Phishing
XR devices and applications are susceptible to malware attacks that can compromise user data, disrupt experiences, or hijack device functionality. This includes deploying ransomware through which access to a user’s AR device is gained and the user’s behavior and interactions in the AR environment are recorded and exfiltrated.

XR environments can be targeted by sophisticated phishing schemes that exploit the immersive nature of the technology. Phishing attacks can trick users into divulging sensitive information or credentials, leading to data breaches or financial loss.

Identity Theft and Impersonation
XR systems often collect personal and biometric data that can be valuable targets for identity thieves. Unauthorized access to this data can result in identity theft, financial fraud, and privacy breaches. Malicious actors can also impersonate legitimate users or entities, leading to trust issues and the manipulation or exploitation of users. If cybercriminals obtain motion-tracking data from a VR headset, they could create digital doubles (known as deepfakes), thus jeopardizing VR security.

Data Integrity and Tampering
Ensuring data integrity in XR applications is critical, as tampering can alter user experiences, lead to misinformation, or cause operational disruptions. Malicious actors can tamper with XR content or data streams, potentially altering experiences or injecting harmful content.

Vulnerabilities in XR Hardware and Software
XR devices can have hardware vulnerabilities that, if exploited, can lead to data breaches, unauthorized access, or device malfunction. In addition, XR applications and platforms can contain software vulnerabilities that expose them to exploits, including remote code execution, data leakage, and denial of service (DoS) attacks.

Social Engineering
Due to inconsistency or unreliability in their processing and results, AR systems can be leveraged for social engineering attacks. Cybercriminals could alter users’ perceptions by presenting false information that deceives them into taking actions that serve the hackers’ interests.

Denial of Service
DoS disrupts XR service availability and the continuous flow of critical information streaming via the services users rely on, leading to significant interruptions and potential safety hazards. For example:

• An AR helmet used by construction workers for overlaying blueprints and real-time safety alerts could result in workers losing access to critical construction plans and safety warnings.
• An AR windshield system in a vehicle could result in the driver losing all augmented navigation and safety information, potentially leading to accidents.

Governance and Regulation of Extended Reality

The rapid evolution of XR technologies presents unique governance and regulatory challenges. Ensuring that these technologies are developed and used responsibly requires a robust regulatory framework, industry self-regulation, and international cooperation.

The regulatory landscape for XR technologies is still developing. While some existing regulations apply to certain aspects of XR, such as data privacy laws including the EU General Data Protection Regulation (GDPR)³ or the US Health Insurance Portability and Accountability Act (HIPAA),⁴ there is no comprehensive regulatory framework specifically tailored to XR. Different industries using XR may be subject to their own sector-specific regulations (e.g., imposed by the U.S. Food and Drug Administration [FDA] in healthcare or the Society of Automotive Engineers [SAE] for AR-enhanced navigation systems).

Regulations often lag behind technological advancements, leading to gaps and inconsistencies in coverage. Furthermore, XR applications often span multiple jurisdictions, creating challenges for consistent regulation and enforcement. The global nature of XR technologies means that regulations must be harmonized across jurisdictions to be effective. Regulators must balance the need to protect consumers and ensure safety without stifling innovation and development in the XR industry.

Developing international standards for XR technologies can ensure consistency and interoperability across different jurisdictions and platforms. Some of the key initiatives in this space include:

• The International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) 5927:2024 standard⁵ governs augmented and virtual reality safety.
• The Institute of Electrical and Electronics Engineers (IEEE) Global Initiative on Ethics of Extended Reality⁶ invites collaboration with subject matter experts from diverse industry and policy backgrounds to create white papers, workshops, and other resources.
• Some privacy and data protection concerns can be mitigated from the outset by implementing Article 25 of the EU GDPR,⁷ which emphasizes data protection by design and default.
• XR Safety Intelligence has been developing the X Reality Safety Intelligence (XRSI) Privacy Framework,⁸ with a baseline set of standards, guidelines, and best regulation-agnostic practices by incorporating privacy requirements from the GDPR, the US National Institute of Standards and Technology (NIST) framework, the US Family Educational Rights and Privacy Act (FERPA),⁹ and the US Children’s Online Privacy Protection Rule (COPPA).¹⁰

As such, the legal vetting of XR technology requirements should consider the protection of intellectual property rights, risk of infringement on existing intellectual property, business liabilities, what and how data privacy laws can be applied, what type of consumer protection laws are applicable, and the implications of using XR in public spaces.

The governance and regulation of XR technologies require a multifaceted approach that includes robust regulatory frameworks, proactive industry self-regulation, and international collaboration. By addressing the unique challenges posed by XR and leveraging opportunities for global cooperation, it is possible to ensure that these technologies are developed and used in a way that is safe, ethical, and beneficial for all.

Case Study

A healthcare technology organization intends to revolutionize medical practices using XR. It develops a product platform designed to facilitate remote surgeries, advanced diagnosis, and virtual immersive consultations by leveraging VR, AR, and MR technologies. An indicative architecture for this use case consists of these components:

• User Interface Layer
  • VR headsets—For immersive remote surgeries and consultations
  • AR glasses—For overlaying diagnostic data on physical objects
  • MR devices—For combining real and virtual worlds in diagnostic and consultative scenarios
• Application Layer
  • Health management app—Core application handling the XR interactions, real-time data processing, and user management
  • Analysis module—Provides real-time analytics and AI insights based on medical data
• Data Layer
  • Patient data repository—Securely stores patient records, medical images, and XR session data
  • Integration of application programming interfaces (APIs)—Interface with existing hospital systems (electronic health records [EHR]/electronic medical records [EMR])
• Network Layer
  • 5G/WiFi—Ensures high-speed, low-latency communication
  • Secure virtual private network (VPN)—For safe data transmission between devices and servers

A safe and secure strategy with due diligence to implement this use case should account for a broad range of ethical, regulatory/compliance, privacy, and security considerations.

Ethical Considerations
Ethical considerations are critical when developing an XR-based healthcare system. Ensuring patient consent and maintaining confidentiality are paramount, especially given the sensitivity of medical data handled by immersive technologies. Engaging in transparent communications about how data will be used and obtaining informed consent for XR sessions are essential. It is important to ensure accessibility and avoid bias in AI analytics to provide equitable healthcare services. Additionally, ethical considerations must address the potential psychological impacts of immersive technologies on patients, ensuring they are not subjected to distress or disorientation during their use.

Regulatory/Compliance Considerations
Regulatory compliance is crucial for the safe and legal operation of the XR healthcare platform. Compliance with HIPAA is necessary to protect patient information in the United States, while EU GDPR compliance is required for handling data from international users. Adhering to medical device regulations, such as the US FDA guidelines for XR equipment and software,¹¹ ensures that the technology meets safety and efficacy standards. Secure data transmission must be maintained through robust cybersecurity measures to prevent breaches. Integration with existing hospital systems must comply with industry standards for data interoperability and security, ensuring seamless and secure data exchanges.

Privacy Considerations
The built-in sensors in user interface devices (VR headsets, AR glasses, MR devices) collect patients’ health data and doctors’ surgical data while the devices are in use or even afterward. The consequences of having such sensitive data leaked to the wrong hands can be life-threatening. Privacy policies must be established that cover, but are not limited to:

• Type of data collected—Medical records, illness details, surgery details, real-time health parameters
• Purpose of data collected—Better diagnosis, timely detection of ailments and emergencies, accurate treatments, prediction of future health hazards
• Mode of data collection—User-supplied data, sensors collected and provided by vendors/third parties
• Storage locations—Real-time data in devices, content delivery (CDN)/edge networks, health management applications, cloud-based analysis engines
• Protection mechanisms—Encryption in transit from devices to CDN and cloud, encryption at rest, access control, logging, approval workflows
• Data sharing—Surgeons, medical consultants, pharmacists, third parties
• Data rights—Update, limit, delete, deny/withdraw consent, and notification modes/channels

Security Considerations
Understanding the attack vectors targeting the healthcare system is crucial for safeguarding user data and ensuring secure interactions. These vectors can exploit vulnerabilities in devices, patient privacy, and user interfaces, making it essential for organizations to anticipate and mitigate potential threats. Vectors include:

• Attacks targeting user interface layer devices include malicious actors planting malware to exfiltrate or corrupt health and medical data, tracking patient movements and telemetry data, harvesting or stealing surgeon credentials that authenticate to health apps, motion-based side channel attacks via sensors, and exploiting behavioral authentication.
• Attacks targeting patients’ privacy include impersonations or deepfakes of patients/surgeons using the data collected via sensors, movements, vibrations, and AR glasses, and usage of continuously active cameras and environmental models to capture users’ surroundings and record bystanders.
• Attacks targeting the patients or surgeons include exploiting boundary systems to make users think they are in safe areas (such as clinics or diagnosis centers), manipulating the overlay content¹² to display false health information or hide real data, and controlling movements by modifying visual signals in AR devices, which can mislead doctors or patients into making incorrect decisions based on manipulated information.

Protection Mechanisms
To effectively protect the healthcare XR system from emerging security threats, it is essential to implement comprehensive protection mechanisms. These safeguards address vulnerabilities across various layers of XR technology, ensuring the integrity, privacy, and security of user data and interactions:

• Device security—Trusted platform module, drive encryption, secure boots, kernel protection, data and write protection
• Account security—Knowledge-based authentication, biometric authentication, behavior-based biometric authentication, unlock patterns, QR codes, code verification through out-of-band channels, multifactor authentication (MFA) with two or more of these methods
• Application security—Account activity monitoring, detect and prevent policy violations, measures to counter phishing via third-party websites, security updates, and patches
• Communication security—End-to-end encryption in transit, mutual authentication, geo- and anomaly-based validation
• Data security—Anonymization, pseudonymization, homomorphic encryption, encryption at rest, sensitive data isolation 

Organizations must be vigilant about the risk associated with XR systems, especially in sensitive fields such as healthcare. Compromises in security can lead to serious consequences, such as patient harm, data breaches, or misdiagnosis. Implementing robust protection mechanisms is essential to safeguard both patient safety and organizational integrity and ensure that XR technologies are used effectively and securely.

Secure Implementation Guidelines

Implementing XR solutions requires a comprehensive approach to ensure not only technical robustness and resilience, but also the safety, privacy, and satisfaction of users. A checklist highlighting the critical aspects can be used to guide practitioners in deploying secure and effective XR systems:

1. Define objectives and business use cases.
   • Identify business use cases by mapping out specific business needs and objectives that XR technology could address.
   • Evaluate and select the XR technology suitable for the identified use case(s). For example, VR can be a fitting solution for immersive training and simulations, while AR can potentially enhance real-time operations with contextual data overlays.
   • Establish success metrics and performance indicators that align with business objectives. This could range from user engagement levels to the effectiveness of training programs or operational efficiency improvements.
2. Design robust architecture.
   • Integrate core applications, real-time data processing, analytics modules, and connectivity solutions into the XR system.
3. Implement data protection mechanisms.
   • Secure data storage with encryption for data at rest and in transit.
   • Anonymize or tokenize personal identifiable information (PII) or sensitive data.
   • Integrate APIs and interfaces with existing systems securely and regularly update security protocols.
4. Manage PII.
   • Obtain informed consent from users before collecting any PII and clearly disclose how their data will be used throughout the life cycle.
   • Implement and enforce a thorough process for managing the PII life cycle, including access control, data retention, and secure deletion.
5. Ensure human safety.
   • Design XR experiences to minimize the risk of physical discomfort (e.g., motion sickness, eye strain, accidental injury).
   • Provide clear safety guidelines and emergency procedures for users operating XR devices.
6. Enhance and secure communication channels.
   • Use high-speed, low-latency network channels, 5G, fiber optics, and high-bandwidth WiFi and CDNs to support the intensive data requirements of XR applications.
   • Employ ring-fencing of XR infrastructure and applications, implement mutual authentication for all critical data transmission, and use firewalls to block unauthorized access and monitor traffic for suspicious activity.
7. Implement robust device and application security measures.
   • Equip devices with trusted platform modules, secure boot processes, and encryption.
   • Use multifactor authentication (MFA), biometric authentication, and continuous account activity monitoring for user accounts.
8. Adhere to industry regulations.
   • Identify and implement controls per any industry standards and regulations relevant to the enterprise and its XR infrastructure.
   • Perform regular reviews and update compliance policies to align with evolving regulations and industry best practices.
9. Integrate ethical practices.
   • Ensure XR technology is used responsibly, with transparent data usage policies and user consent.
   • Address ethical concerns pertaining to the impact of immersive experiences on users, which includes data manipulation or misinformation.
10. Perform end-to-end testing.
    • Perform periodic security audits, fuzz testing, vulnerability assessments, and penetration testing to identify and mitigate risk.
    • Conduct end user acceptance testing to validate the functionality, usability, and user experience of the XR ecosystem.
11. Provide training and collaboration services.
    • Offer training programs that cover XR device operation, safety aspects, and best practices for effective and safe interaction.
    • Promote collaboration among teams involved in the development, deployment, and management of XR solutions.
    • Provide instruction materials that encompass the security, privacy, and ethics of XR.
12. Implement support systems.
    • Develop a support infrastructure for troubleshooting, optimizing, and maintaining XR systems.
    • Provide regular updates and patches to address security vulnerabilities and enhance performance.
13. Monitor and assess performance.
    • Implement monitoring tools to continuously track the XR infrastructure (devices, applications), performance, security incidents, and user feedback.
    • Use performance metrics and logs to assess the effectiveness of XR applications and identify areas for improvement.
14. Facilitate data access and recovery.
    • Implement data redundancy mechanisms and backup strategies to safeguard against data loss.
    • Develop data recovery plans to restore operations quickly in the event of a failure or breach.
15. Review and update policies.
    • Perform periodic reviews and updates of the security, privacy, and ethical policies to reflect new threats, technology advancements, and regulatory changes.
    • Conduct periodic audits and assessments to ensure ongoing compliance and risk management.

Adopting this checklist will help practitioners implement XR solutions that are secure, ethical, and user centric. By addressing the technical, privacy, and safety considerations, organizations can create XR experiences that are reliable and trustworthy.

Conclusion

XR technologies, including VR, AR, and MR, offer transformative potential across industries such as healthcare, education, and finance. However, to fully realize their benefits while minimizing risk, it is essential to implement robust cybersecurity measures and data privacy while addressing ethical considerations such as informed consent, user privacy, and inclusivity. These are crucial in protecting sensitive user information and fostering responsible XR use.

Adherence to regulatory standards and industry specific compliance requirements is vital for safeguarding user data and ensuring the safety and effectiveness of XR technologies. Industry collaboration on international standards and best practices can further enhance the ethical use of XR. Proactively addressing these challenges will allow the innovative potential of XR technologies to be harnessed while ensuring their secure and responsible integration into the human experience.

Endnotes

¹ Virtual Reality Society, “History of Virtual Reality” 
² Ohai, F.; “The Threat Landscape of Extended Reality: Beyond Privacy and Data Protection,” Ku Leuven CiTiP Blog, 21 November 2023
³ Intersoft Consulting, “General Data Protection Regulation (GDPR)” 
⁴ U.S. Department of Health and Human Services, “Summary of the HIPAA Privacy Rule,” USA
⁵ International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC), ISO/IEC 5927:2024 Computer graphics, image processing and environmental data representation —Augmented and virtual reality safety —Guidance on safe immersion, set up and usage, 2024
⁶ Institute of Electrical and Electronics Engineers (IEEE) Standards Association, “The IEEE Global Initiative on Ethics of Extended Reality” 
⁷ Intersoft Consulting, “Art 25 GDPR Data Protection by Design and by Default” 
⁸ XRSI, “The XRSI Privacy Framework” 
⁹ U.S. Department of Education, “Family Educational Rights and Privacy Act (FERPA),” USA
¹⁰ Federal Trade Commission, “Children’s Online Privacy Rule (COPPA),” USA
¹¹ U.S. Food and Drug Administration, “Augmented Reality and Virtual Reality in Medical Devices,” USA
¹² Casey, P.; Baggili, I.; Yarramreddy, A.; “Immersive Virtual Reality Attacks and the Human Joystick,” IEEE, 2019

BALAJI SWAMINATHAN

Is an experienced security architect with extensive experience designing and implementing robust security solutions to safeguard digital environments and ensure the resilience and integrity of diverse IT infrastructure. Currently he is a senior product security architect overseeing the security of cloud services, content delivery, and products.