arXiv pre-print View it on Github Demo

An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception

TL;DR: REALM maps event-camera data into the frozen latent space of the DUNE RGB foundation model using lightweight LoRA adapters — no task-specific training. This lets a single encoder drive depth estimation and segmentation via simple linear heads, and enables the direct, zero-shot use of frozen image-trained decoders like MASt3R on raw event data, setting a new state of the art in event-based feature matching.

Vincenzo Polizzi¹, David B. Lindell², Jonathan Kelly¹

¹University of Toronto, Robotics Institute
²University of Toronto, Department of Computer Science

Abstract

By mapping a sparse event stream into DUNE's latent space, REALM enables diverse downstream applications: zero-shot feature matching and 3D reconstruction with the frozen MASt3R decoder, and dense prediction tasks like semantic segmentation and depth estimation using simple linear heads.

Highlights

Method

REALM builds on DUNE, a universal ViT encoder distilled from heterogeneous RGB and 3D teachers (including MASt3R). We replace DUNE's image patch embedder with a lightweight convolutional stem that maps a 5-bin event voxel grid into the same token format, and adapt the frozen backbone to the event modality with Low-Rank Adaptation (LoRA) — leaving the pretrained weights untouched to avoid catastrophic forgetting. Training uses a dual-masking distillation strategy: a progressive spatial mask restricts the distillation loss to regions with active events early in training (to avoid hallucinating static backgrounds the event stream never observed) and gradually dilates outward to encourage holistic scene understanding, while MAE-style patch dropout forces the encoder to build compressed, context-aware representations rather than relying on local interpolation.

Event representations undergo MAE-style patch dropout before being processed by a trainable embedding layer and a LoRA-adapted student encoder. The network is trained to match the latent representation of the frozen DUNE teacher, with a progressive spatial mask focusing the distillation loss on regions with active event data.

Spatial feature maps (PCA) show that REALM implicitly segments objects and identifies semantic structures similarly to the RGB teacher, while the masking strategy avoids representing the static background that is invisible to the events. The UMAP plot (right) confirms this alignment, showing consistent clusters where event features sit close to their RGB counterparts — enabling zero-shot transfer of downstream tasks.

Interactive Demo

Try REALM on your own data. Upload an RGB image and an event voxel grid (see github repo on how to obtain it).

Qualitative Results on Downstream Tasks

+

+

+

×

Monocular Depth Estimation

REALM preserves dense scene structures using a simple linear projector evaluated on MVSEC sequences, showcasing strong robustness in challenging nighttime conditions where frame-based sensing degrades.

RGB Events GroundTruth REALM

×

Dense Semantic Segmentation

REALM effectively identifies key classes (e.g., road, vehicles, pedestrians) on the DSEC driving dataset using a frozen backbone and a single linear head, inheriting strong scene understanding directly from the RGB teacher.

RGB Events GroundTruth REALM

×

Cross-Modal Feature Matching

Zero-shot cross-modal (RGB-Event) and intra-modal (Event-Event) matching leveraging the frozen MASt3R decoder. REALM operates natively in a geometrically consistent latent space to produce robust correspondences under wide viewpoints.
In green are the correct matches in orange the incorrect ones (given by RANSAC).

Quantitative Results

REALM uses a single frozen encoder across all three tasks: linear heads trained on RGB/DUNE features are transferred zero-shot to event features for depth and segmentation, and the frozen, RGB-trained MASt3R head is used as-is for feature matching — no event-specific fine-tuning of any decoder.

Wide-Baseline Feature Matching (AUC, %)

REALM outperforms all specialized event-based matching baselines by a wide margin on both the ECD and EDS datasets, despite using a frozen, image-trained MASt3R head with no event fine-tuning.

Monocular Depth Estimation — Avg. Absolute Error [m] (MVSEC)

Bold = best, underline = second-best among event-based methods (DUNE is an RGB reference, not a competing baseline). REALM wins or ties for best on 3 of 4 nighttime/outdoor sequences and, most notably, consistently outperforms its own RGB teacher, DUNE, at night — highlighting the inherent advantage of event-based sensing in high-dynamic-range conditions.

Semantic Segmentation (DSEC, 11 classes)

Using only a single linear head transferred zero-shot from the frozen DUNE encoder, REALM is competitive with specialized, purpose-built event-segmentation architectures — without any edge-guidance modules or recurrent decoders.

Inference Efficiency (vs. MINIMA, per image pair)

REALM is roughly 2× faster and uses under a third of the GPU memory of MINIMA for cross-modal matching, while achieving higher matching accuracy.

Cite this work

@inproceedings{polizzi_2026_realm,
      title={REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception}, 
      author={Vincenzo Polizzi and David B. Lindell and Jonathan Kelly},
      booktitle={European Conference on Computer Vision (ECCV)},
      year={2026}
}