Ray-SCR: Generalizing Scene Coordinate Regression Across Camera Models

Jaemin Cho¹Oleksii Nasypanyi¹Francois Rameau²

¹ Stony Brook University² SUNY Korea

jaemin.cho@stonybrook.edu oleksii.nasypanyi@stonybrook.edu francois.rameau@sunykorea.ac.kr

WACV 2027

arXiv Paper Code

Page under construction (Stay tuned!)

Input image

Encoder

Ray Adapter

SCR Backbone

[1]ACE-G. Leonard Bruns et al. ACE-G: Improving Generalization of Scene Coordinate Regression Through Query Pre-Training. ICCV, 2025.
[2]ACE. Eric Brachmann et al. Accelerated coordinate encoding: Learning to relocalize in minutes using rgb and poses. CVPR, 2023.
[3]GLACE. Fangjinhua Wang et al. Glace: Global local accelerated coordinate encoding. CVPR, 2024.
[4]DSAC*. Eric Brachmann et al. Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC. TPAMI, 2022.

Abstract

Among visual localization strategies, scene coordinate regression (SCR) has emerged as an attractive approach for its compactness, accuracy, and speed. Despite these advantages, SCR models generalize poorly across substantially different camera models, such as non-pinhole or strongly rectified imagery. This limitation arises because, changes in field of view, lens distortion, or rectification-induced resampling can make the same scene content appear differently depending on its pixel position, degrading localization accuracy. To cope with this limitation, we introduce Ray-SCR, a lightweight extension to existing SCR architectures that conditions local visual features on their calibrated viewing rays and refines neighboring ray-conditioned features with a shallow local Transformer. We further formulate supervision directly in ray space and use sparse SfM points as metric anchors that link observations across views and camera models. Taken together, these components enable a single scene representation built from a panoramic mapping sequence to accurately localize query images captured with very different cameras, including pinhole, fisheye, and omnidirectional cameras.

We evaluate Ray-SCR across multiple SCR architectures and on datasets containing pinhole, fisheye, and omnidirectional cameras. The proposed ray conditioning substantially improves generalization across cameras and leads to state-of-the-art localization performance in these challenging cross-camera settings.

Method

fig2_mapping_v30.pdf — **Ray-SCR scene-training pipeline.** We first extract encoder features from the training images, optionally augmented with sampled virtual views. SfM points are projected into each view, and features whose angular distance θ_i from the corresponding viewing ray r_i is below a predefined threshold are retained in the training buffer. The Ray Adapter then injects camera-geometry information by encoding the associated viewing rays into the features and locally processing groups of neighboring features with a lightweight Transformer. The adapted features are subsequently passed through the standard SCR pipeline. Here, ACE-G is shown as an example, using a DINO encoder together with trainable scene-specific map codes.

Experiments

BibTeX

@misc{cho2026rayscr,
  title         = {Ray-SCR: Generalizing Scene Coordinate Regression Across Camera Models},
  author        = {Cho, Jaemin and Nasypanyi, Oleksii and Rameau, Francois},
  year          = {2026},
  eprint        = {2611.03098},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2611.03098}
}