VR & AR Development

Extended reality (XR) is the umbrella term for virtual reality (VR), which replaces the user’s view of the world, and augmented/mixed reality (AR/MR), which layers synthetic content onto it. This page is a self-contained reference to the engineering side of XR as of late 2026: the device landscape, how a frame gets from the application to the eye, the runtime and engine stack (OpenXR, Unity, Unreal, native SDKs, WebXR), mobile and headset AR, and the human-factors rules that decide whether an experience is comfortable.

Four constraints shape almost every XR design decision:

Prerequisites. 3D math (vectors, quaternions, homogeneous transforms), a game engine (Game Development Hub), the rendering pipeline (3D Graphics & Rendering) and profiling (Performance Optimization). For phone AR add Swift (ARKit) or Kotlin (ARCore).

Reading paths

The XR Landscape

The reality–virtuality continuum

Milgram and Kishino (1994) described a continuum from the unmodified physical world to a fully synthetic one. Modern devices blur the categories: a passthrough headset such as Quest 3 or Vision Pro spans the whole range in software.

flowchart LR
    R["Real environment<br/>no synthetic content"] --> AR["Augmented reality<br/>overlays registered to the world<br/><i>phone AR, smart glasses</i>"]
    AR --> MR["Mixed reality<br/>virtual content occluded by and<br/>interacting with real geometry<br/><i>Quest 3, Vision Pro, Galaxy XR</i>"]
    MR --> V["Virtual reality<br/>fully synthetic view<br/><i>PC VR, PSVR2, Steam Frame</i>"]

Moving right, the system renders more and relies less on the real world: the engineering problem shifts from world understanding (AR) toward rendering throughput and comfort (VR).

Headsets (late 2026)

Specifications change with firmware; treat the table as orientation, not a procurement sheet. Older devices still common in the field include Quest 2 and Quest Pro (both discontinued), Valve Index and HTC Vive-family PC headsets.

AR glasses and phones

The XR Frame Pipeline

The core engineering loop of any headset is: predict where the head (and eyes, and hands) will be when photons leave the display, render for that pose, then correct the image at the last moment for whatever the prediction got wrong.

sequenceDiagram
    participant S as Sensors (IMU + cameras)
    participant R as XR runtime / compositor
    participant A as Application
    participant G as GPU
    participant D as Display
    A->>R: xrWaitFrame (throttle to display rate)
    R-->>A: predicted display time
    A->>R: xrLocateViews(predicted time)
    R-->>A: predicted eye poses + FOV
    A->>G: record and submit stereo frame
    G-->>R: xrEndFrame (layers: projection, quad, ...)
    S->>R: latest head pose (~1 ms before scan-out)
    R->>G: reproject / timewarp to latest pose, lens distortion
    G->>D: scan-out

Key terms:

• Motion-to-photon latency – time from a head movement to light reflecting it. Pose prediction makes the effective latency far lower than the raw pipeline depth; the target is imperceptible (about 20 ms or less).
• Pose prediction – the runtime extrapolates head motion to the predicted display time it hands the app. Always render with the pose from that call, never a pose you sampled yourself.
• Late latching – some runtimes (Meta Quest) patch the pose into GPU constant buffers after the CPU has recorded commands, shaving a frame of latency.
• Compositor – the runtime process that owns the display. Apps submit layers; the compositor applies lens distortion and chromatic-aberration correction, reprojection, and draws system UI.
• Compositor layers – text and UI submitted as separate quad or cylinder layers are sampled once at display resolution by the compositor, so they stay sharper than the same content rendered into the eye buffer.

Frame budgets

CPU and GPU work are pipelined (the GPU renders frame N while the CPU simulates N + 1), so each must fit the budget independently rather than summed. On standalone hardware leave headroom: thermal throttling and the compositor itself consume part of the GPU.

Reprojection and frame synthesis

When an app misses its deadline the runtime shows a corrected old frame rather than a stutter. The techniques differ in what they can correct:

All frame synthesis produces artifacts in disoccluded regions (content revealed behind moving objects), on transparent surfaces and on fast-moving thin geometry. Treat it as a safety net or a deliberate budget tool, not a substitute for hitting frame rate.

Rendering for XR

Stereo rendering

Each eye needs its own view and projection matrix; the naive approach draws the scene twice.

Engines expose this as a setting (Unity Render Mode: Single Pass Instanced / Multiview, Unreal Instanced Stereo / Mobile Multi-View). The GPU still shades both eyes’ pixels, which is why pixel-side techniques matter more.

Foveated rendering

The eye resolves fine detail only in the central ~2° of the fovea; acuity falls off steeply outside it. Foveated rendering spends shading work accordingly.

Mechanisms include variable-rate shading (VRS), fragment density maps (Vulkan) and Metal rasterization-rate maps. Savings depend on how pixel-bound the app is; eye-tracked foveation gives larger savings than fixed because the full-rate region can be much smaller. Eye-tracking latency and saccades are the limiting factor: the high-detail region must be large enough to cover gaze-prediction error.

Performance optimization

Profiling tools: OVR Metrics Tool and Meta Quest Developer Hub (Quest), RenderDoc and vendor GPU profilers (Snapdragon Profiler, Arm Performance Studio), Xcode Instruments / RealityKit Trace (visionOS), Unity Profiler and Unreal Insights, PIX on Windows PC VR.

OpenXR and the Runtime Stack

OpenXR (Khronos) is the cross-vendor API between applications and XR runtimes. Version 1.0 shipped in 2019 and 1.1 in April 2024, which folded several widely used extensions into core; the SDK now receives roughly monthly 1.1.x revisions. Meta, Valve (SteamVR), Pico, HTC, Magic Leap, Microsoft and Google’s Android XR all ship conformant OpenXR runtimes, and Unity, Unreal and Godot use OpenXR as their primary XR backend. Apple’s visionOS is the notable exception: it uses its own ARKit/RealityKit/Compositor Services APIs.

flowchart TB
    subgraph App["Application layer"]
        U["Unity<br/>XR Interaction Toolkit / AR Foundation"]
        UE["Unreal Engine<br/>OpenXR plugin"]
        GD["Godot / native C++"]
        WX["WebXR page<br/>(three.js, Babylon.js)"]
    end
    BR["Browser<br/>(Chromium, Quest Browser)"]
    L["OpenXR loader"]
    subgraph RT["Vendor runtimes"]
        M["Meta Horizon OS"]
        SV["SteamVR / SteamOS"]
        AX["Android XR"]
        OT["Pico, Magic Leap, ..."]
    end
    V["visionOS<br/>ARKit + RealityKit + Compositor Services"]
    U --> L
    UE --> L
    GD --> L
    WX --> BR --> L
    L --> M & SV & AX & OT
    U -. "PolySpatial / visionOS plugin" .-> V

Core OpenXR concepts:

Session state machine — apps must only render and submit frames in the states the runtime allows:

stateDiagram-v2
    [*] --> IDLE: xrCreateSession
    IDLE --> READY: runtime ready
    READY --> SYNCHRONIZED: xrBeginSession
    SYNCHRONIZED --> VISIBLE: frames shown
    VISIBLE --> FOCUSED: receives input
    FOCUSED --> VISIBLE: system UI takes focus
    VISIBLE --> SYNCHRONIZED: hidden
    SYNCHRONIZED --> STOPPING: runtime requests stop
    STOPPING --> IDLE: xrEndSession
    IDLE --> EXITING: app or runtime quits
    EXITING --> [*]: xrDestroySession

Engines and SDKs

Input Modalities

Design principles: provide a ray for far interaction and direct touch/poke for near interaction; give every interaction visual (and where possible haptic or audio) feedback on hover, press and release; never require precision that hand tracking cannot deliver (targets smaller than about 1.5–2 cm are unreliable).

Hand and body tracking

OpenXR and WebXR report 26 hand joints (palm, wrist, and four or five joints per finger including tips) with position, orientation and radius. Gesture recognition is usually done in the app or SDK from joint poses (pinch strength from thumb–index tip distance). Full-body avatars on consumer headsets are mostly inferred: upper body from head and hand poses via inverse kinematics, legs synthesised by learned models (Meta’s Movement SDK generates legs on Quest 3) rather than directly tracked.

Locomotion and Comfort

Why XR makes people sick

Simulator sickness is mostly a visual–vestibular conflict: the eyes report self-motion that the inner ear does not. Linear and especially angular acceleration are the strongest triggers; constant-velocity motion is tolerated better. Sensitivity varies widely between users and decreases with exposure.

A second, independent discomfort source is the vergence–accommodation conflict (VAC): the eyes converge on the virtual depth of an object while focusing (accommodating) on the fixed optical focal plane of the display, typically 1–2 m. Content far from the focal distance, especially very close to the face, causes eye strain. Varifocal and light-field displays remain research and prototype technology.

Locomotion options

flowchart TD
    Q{"Does the experience need<br/>artificial movement?"} -->|No| RS["Room-scale / physical walking<br/>(most comfortable)"]
    Q -->|Yes| C{"Comfort priority?"}
    C -->|Maximum| T["Teleport + snap turn<br/>(no visual acceleration)"]
    C -->|Balanced| S["Smooth move + snap turn<br/>+ tunnelling vignette"]
    C -->|Immersion, experienced users| F["Smooth move + smooth turn<br/>(offer as opt-in)"]
    T & S & F --> O["Always: user-selectable options,<br/>seated mode, stable horizon"]

Spatial UI

Placement and sizing

Size UI by visual angle, not by metres or pixels. An element subtending angle $\theta$ at distance $d$ has height

so body text at about 1° at 1.5 m is roughly 2.6 cm tall; minimum legible sizes depend on the headset’s pixels per degree.

Anchoring

Accessibility

Provide seated and standing modes with height calibration, one-handed and remappable controls, alternatives to every motion or gesture requirement, subtitles positioned in world space with speaker direction indicators, colour-blind-safe indicators, adjustable text size, and head-gaze fallbacks for eye or hand tracking. The XR Accessibility User Requirements (W3C) and platform guidelines (Meta, Apple Human Interface Guidelines for visionOS) are the standard references.

AR Development

How tracking works

Phones and headsets both use visual–inertial odometry (VIO): high-rate IMU integration gives smooth short-term motion, and camera feature tracking corrects the drift. Building and re-localizing against a map of visual features (SLAM) lets content stay in place across the session.

flowchart LR
    IMU["IMU<br/>~1 kHz accel + gyro"] --> F["Sensor fusion<br/>(VIO filter / optimizer)"]
    CAM["Camera frames<br/>30-60 Hz feature tracks"] --> F
    DEP["Depth<br/>(LiDAR / ToF / stereo)"] --> SU
    F --> POSE["6DoF device pose"]
    F --> MAP["Feature map<br/>(relocalization, anchors)"]
    CAM --> SU["Scene understanding<br/>planes, mesh, semantics, depth"]
    POSE --> REN["Renderer"]
    SU --> REN
    MAP --> ANC["Anchors"] --> REN

Tracking degrades on textureless surfaces (white walls), in low light, with fast motion blur, and in dynamic scenes; good apps surface tracking state (“move your device slowly”) instead of failing silently.

Scene understanding

ARKit vs ARCore

Spatial anchors and shared space

An anchor pins a pose to features of the real environment so the runtime can keep correcting it as its map improves. Rules: anchor content near where it was placed (error grows with distance from the anchor), use one anchor per cluster of content rather than one per object, and never assume anchor poses are fixed between frames.

Microsoft Azure Spatial Anchors was retired in November 2024; apps that depended on it must migrate to platform-native or vendor anchor services.

Passthrough mixed reality

Video-passthrough headsets (Quest 3, Vision Pro, Galaxy XR, Steam Frame) reconstruct the real world from cameras and reproject it to the eyes, which allows full occlusion and relighting but adds latency and distortion that optical see-through (HoloLens, Magic Leap) avoids. MR apps on these devices combine passthrough with scene data (room mesh, planes) so virtual objects sit on real tables and are hidden by real furniture. Raw camera access for computer vision is now available with user permission on some platforms (Meta’s Passthrough Camera API on Quest 3/3S, Android XR camera access, visionOS enterprise entitlements); otherwise apps receive only derived data.

WebXR

The WebXR Device API exposes immersive-vr and immersive-ar sessions to web pages. It is supported in Chromium browsers (Chrome/Android, Edge, Quest Browser, Android XR) via OpenXR, and in Safari on visionOS for immersive-vr; iOS Safari does not support WebXR. Feature modules add hand input, hit testing, anchors, plane detection, depth sensing, layers and DOM overlays; availability varies per browser. Sessions must be started from a user gesture (a click on an “Enter VR” button).

A minimal session with WebGL:

const canvas = document.createElement('canvas');
const gl = canvas.getContext('webgl2', { xrCompatible: true });
let refSpace;

async function enterXR() {
  if (!navigator.xr || !(await navigator.xr.isSessionSupported('immersive-vr'))) {
    return; // show a non-XR fallback
  }
  // Must be called from a user-activation event handler.
  const session = await navigator.xr.requestSession('immersive-vr', {
    requiredFeatures: ['local-floor'],
    optionalFeatures: ['hand-tracking', 'bounded-floor'],
  });
  session.updateRenderState({ baseLayer: new XRWebGLLayer(session, gl) });
  refSpace = await session.requestReferenceSpace('local-floor');
  session.addEventListener('end', () => { refSpace = null; });
  session.requestAnimationFrame(onXRFrame);
}

function onXRFrame(time, frame) {
  const session = frame.session;
  session.requestAnimationFrame(onXRFrame);

  const pose = frame.getViewerPose(refSpace);
  if (!pose) return; // tracking lost this frame

  const layer = session.renderState.baseLayer;
  gl.bindFramebuffer(gl.FRAMEBUFFER, layer.framebuffer);
  gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);

  for (const view of pose.views) {           // one view per eye
    const vp = layer.getViewport(view);
    gl.viewport(vp.x, vp.y, vp.width, vp.height);
    drawScene(view.projectionMatrix, view.transform.inverse.matrix);
  }
}

document.querySelector('#enter-vr').addEventListener('click', enterXR);

In practice most WebXR work uses three.js (renderer.xr), Babylon.js (WebXRDefaultExperience) or A-Frame, which wrap this loop, controller input and hit testing.

Development Workflow

Emerging Directions

• Display glasses and AI assistants. Lightweight glasses with monocular or binocular HUDs (Meta Ray-Ban Display, Android XR glasses partners) shift design toward glanceable, context-aware UI driven by multimodal AI rather than full 3D scenes.
• Neural and EMG input. Wrist-worn surface EMG (Meta Neural Band) reads motor-neuron signals for subtle finger gestures without cameras.
• Eye tracking as a default. Eye-tracked foveated rendering and streaming, gaze-based targeting and social eye contact are standard on higher-end headsets and spreading downward.
• Open cross-vendor spatial APIs. Ratified OpenXR spatial-entity extensions (planes, anchors, markers, persistence) are replacing vendor-specific scene APIs.
• Display research. Varifocal and holographic optics to resolve VAC, and wider fields of view in thinner pancake and waveguide optics.

See Also

• Performance Optimization — profiling and frame-budget work; see GPU optimization and CPU optimization for hitting 90–120 Hz
• 3D Graphics & Rendering — the rasterization pipeline behind stereo, multiview and foveated rendering
• Shaders — variable-rate shading and GPU programming
• Game Development — engine fundamentals for interactive XR; UI design and audio design for spatial UI and sound
• Unreal Engine — UE5 VR development with the forward renderer and instanced stereo