LOCI: Leveraging Semantic Maps for City-Scale Cross-View Localization

The problem

Robots lean on prior maps for context about their environment, but a map is only useful if the robot can localize itself within it. Maps built from prior overhead data, like satellite imagery and metric-semantic maps (MSMs) such as OpenStreetMap, carry a representation gap: a street’s name in the map, or how it looks from above, is nothing like the pixels of a ground-level panorama. GNSS could sidestep the gap by providing the robot’s position directly in the overhead map, but it is unreliable without clear line of sight to the satellites (urban canyons, dense tree cover) and is susceptible to jamming.

Image-based cross-view localization approaches train models in a contrastive framework to bridge this representation gap between ground photos and satellite patches. However, overhead imagery alone can be ambiguous: city blocks are often self-similar from above, egocentric features can disappear under heavy tree cover, and highly discriminative features like street and business names are not captured in the overhead imagery because of the perspective transform. MSMs record instance-level details that hold promise to break this ambiguity. Existing methods that localize against MSMs often fail to scale to large overhead areas or rasterize the map into a handful of coarse classes and throw the instance-level information away. LOCI leverages instance-level semantic details in a manner that scales to large overhead areas, complementing image matching where images alone fall short.

A quadruped robot on a suburban road. Insets show ambiguous satellite imagery, street name signs, and an OpenStreetMap fragment used to localize the robot. — **Fig. 1.** Self-similarity or domain shift can make satellite imagery alone difficult to localize against (a). Semantic features like street names are often more discriminative and more robust to visual change (b). LOCI extracts these features from the robot’s panoramas and uses them to more rapidly localize within prior overhead data (c).

The approach

LOCI treats localization as Bayesian filtering: a histogram over the region is updated with odometry and with an observation likelihood that is a product of experts, combining an image-matching expert (Wide Area Geolocalization, WAG) with a semantic-map expert. The semantic expert runs in three stages:

1. A VLM reads the scene. Each panorama is rendered into four pinhole views and passed to a vision-language model (Gemini 3 Flash) with a structured prompt that asks for landmarks in OpenStreetMap’s own key=value vocabulary, like {shop: mall, name: Metreon}, so extracted landmarks and mapped landmarks live in the same representation.

A street-level panorama at 4th and Mission Streets in San Francisco, with boxes around the Metreon, Moscone Center West, and the Children's Creativity Museum. — **Fig. 2.** Landmarks a VLM extracts from a San Francisco panorama in our dataset. Boxes mark where the VLM localized each landmark. Salient, mappable objects are kept. Traffic signals, trees, and buildings described only by their appearance are ignored.

2. A lightweight classifier does the matching. Matching extracted and mapped landmarks can be difficult: variations in tag conventions, omissions of certain attributes, and mistakes in reading signs can make it so two landmark descriptions of the same object have no overlap. VLMs excel at disambiguating if sets of tags could describe the same object, but querying the VLM for every extracted-landmark and map-landmark (e.g., 162k+ mapped landmarks in the Boston dataset) pair would be expensive and slow. Instead, the VLM serves as an offline teacher and labels a one-time training set of matches and hard negatives. A lightweight classifier distilled from those labels (0.976 validation ROC-AUC) scores all pairs in about 3.7 s per panorama, roughly two orders of magnitude faster than prompting a local LLM online.

3. Matches become a likelihood. Per-patch correspondence scores are assigned with the Hungarian algorithm, weighted so that distinctive landmarks (a named restaurant) count more than generic ones (a building), and converted into a Gaussian observation likelihood fused with the image expert in the filter.

The dataset

Existing cross-view benchmarks are largely fair-weather, daytime, and image-only. To test if semantics help when images struggle, we assembled eleven environments: Chicago, Seattle, and New York from VIGOR, plus Mapillary and self-collected trajectories covering San Francisco, an active snowstorm and night driving in Boston, rain in Middletown, suburban Framingham, the rural Noordoostpolder and forested Veluwe in the Netherlands, and Fort Myers shortly after Hurricane Ian. The Noordoostpolder, Veluwe, and Fort Myers regions each cover more than 625 km². Every environment is paired with satellite imagery, a dated OpenStreetMap snapshot, VLM-extracted landmark annotations, and for Chicago/Seattle the landmark-correspondence training and validation set.

Representative panoramas, overhead coverage regions, and driven trajectories for the Boston snowy and night, Middletown, Framingham (with its two satellite mosaics), Fort Myers, and Noordoostpolder datasets. — **Fig. 3.** Representative panoramas, overhead coverage (red), and trajectories (yellow) for the snowy and night Boston collects (A), Middletown (B), Framingham (C), Fort Myers post-hurricane (D), and Noordoostpolder (E). Trajectories run from green to red. Orange lines separate the snowy and night panoramas in (A), and the clean MassGIS mosaic (above) from the mixed-date Google mosaic of Framingham Mixed-Sat (below) in (C).

Dataset release is coming soon!

Results

The interactive figure below presents the evaluation results: 5,000 five-kilometer paths through the VIGOR New York panoramas and 1,000 three-kilometer subsets of all other panorama trajectories that were self-collected or sourced from Mapillary. It shows how much probability mass the filter places near the true position as the robot travels, averaged over trajectories. Pick an environment and a radius. Shaded bands are 95% confidence intervals on the mean.

Across most environments, incorporating instance-level semantics meaningfully accelerates convergence. The gains are largest where image-only matching is most strained or landmarks are dense and informative. In landmark-dense cities LOCI converges 3–10 times faster than image-only matching while reaching comparable final error. In Fort Myers, where the trajectory was collected shortly after Hurricane Ian and the satellite imagery predates the storm, LOCI places probability mass near the true position faster than all other approaches while the rasterized-OSM baseline is actively harmful. The clear exception is Veluwe, whose trajectory lies on a landmark-sparse stretch of multi-lane highway through rural areas. The landmarks extracted there are dominantly mile markers, overpasses, and signs referring to places far from the robot, violating the proximity assumption in our observation model, and some, like communication towers, are inconsistently mapped in OSM, pulling probability mass toward the wrong place. There, image-only WAG localizes better.

Framingham probes how much each approach relies on the quality of the overhead imagery. We evaluate the same trajectory against two satellite sources: a clean single-date MassGIS mosaic, and a second mosaic stitched from Google Maps tiles whose underlying imaging dates differ across the region (Framingham Mixed-Sat), leaving visible leaf-on and leaf-off seams. With the clean mosaic every approach converges by the end of the trajectory, though LOCI converges roughly three times faster than WAG. On the mixed mosaic every approach except LOCI fails to gain traction, with EDC₁₀₀ near the trajectory-length ceiling. LOCI does not fully localize either, but its instance-level semantics retain far more localization signal than any other approach.

Expected distance to convergence and final error

Distance traveled weighted by the probability that the belief is more than 100 m from the true location (EDC₁₀₀, lower is better) and final localization error, mean ± 1.96 SEM. Toggle the metric:

Bold = best method for the environment.

Takeaways

Instance-level semantics beat category rasters. Named landmarks and house numbers disambiguate self-similar blocks and survive visual domain shift. Coarse rasterized OSM tiles do neither, and were actively harmful in Fort Myers, Noordoostpolder, and Veluwe, where the rasterized priors did not survive the appearance change or the self-similarity.

One training city, many unseen environments. The correspondence classifier is trained on a single city (Chicago) and validated on one more (Seattle), then run unchanged across snow, night, rain, fog, suburbs, rural farmland, forest, and a post-hurricane disaster zone. Access to both modalities is not sufficient for this transfer. The early-fusion variant (LOCI-EF) failed to match LOCI across most environments, indicating that how image and landmark observations are represented and combined affects whether the filter generalizes beyond its training distribution.

Know where semantics help, and where they don’t. LOCI wins where landmarks are dense, discriminative, and near the robot. On Veluwe’s forest highway the landmarks are dominated by mile markers, overpasses, and signs referring to places far away, violating our observation model’s assumption that observed landmarks are near the robot. There, image-only matching localizes better, marking a clear direction for extending the observation model.

Citation

@article{fahnestockandfuentes2026loci,
  title   = {Leveraging Semantic Maps for City-Scale Cross-View Localization},
  author  = {Fahnestock, Ethan and Fuentes, Erick and Osteen, Philip R. and Roy, Nicholas},
  year    = {2026},
  journal = {arXiv preprint arXiv:2607.25215}
}