Realsee3D Dataset

A large-scale multi-view RGB-D dataset advancing indoor 3D perception, reconstruction, and scene understanding.

View on GitHub Learn More

Overview

10,000 Scenes

Realsee3D is a large-scale multi-view RGB-D dataset designed to advance research in indoor 3D perception, reconstruction, and scene understanding. It contains 10,000 complete indoor scenes, composed of:

1,000 Real-World Scenes Captured by Galois-P4 LiDAR

9,000 Synthetic Scenes Procedurally generated from real floorplans

95,962 Total Rooms Real-world (9,483) + Synthetic (86,479)

299,073 Total Viewpoints Real-world (24,263) + Synthetic (274,810)

Realsee3D aims to serve as a unified benchmark for high-fidelity indoor scene modeling, facilitating research in geometry reconstruction, multimodal learning, and embodied AI.

Beyond RGB-D imagery, the dataset now provides per-viewpoint semantic segmentation (segment.png) for both real-world and synthetic scenes (real-world labels are current-model predictions). Additional annotations — CAD drawings, floorplans, and 3D object detection — are forthcoming.

Real-World Data Acquisition

Our real-world data is collected from indoor residential scenes using the Realsee Galois P4 3D LiDAR scanning system. The system captures high-fidelity synchronized data:

High-Fidelity RGB-D

Unlike traditional panoramic pipelines, our system employs nearly perfectly co-centered cameras and LiDAR. We achieve HDR quality via multi-exposure stacking and ensure precise alignment between depth and color data.

Precise Localization

We use a global registration process that combines visual features and 3D point cloud data to ensure accurate camera poses across the entire scene.

Deep Dive: High-Fidelity Data Processing

High Dynamic Range (HDR) RGB Panorama

Unlike traditional panoramic pipelines that rely on approximated co-centering (e.g., Insta360) or wide-baseline fusion (e.g., Matterport), our system employs nearly perfectly co-centered cameras. This structural advantage significantly simplifies the stitching process and ensures superior physical consistency in the resulting panoramas.

Furthermore, our advanced ISP pipeline guarantees photometric fidelity: stitching is performed directly in the RAW domain for highly consistent color and luminance, and we achieve High Dynamic Range (HDR) quality via multiple-exposure stacking. A sophisticated HDR tone-mapping algorithm preserves this extended range, while the system explicitly handles mixed-illumination conditions, supplying richer, more reliable visual information than most commercial 360° cameras.

LiDAR Depth Data Synthesis

The continuously acquired 3D point cloud data from the LiDAR is post-processed, integrated, and synthesized into a depth image. This depth image is precisely aligned with the panoramic color image via calibrated camera-LiDAR extrinsics. The near-coincidence of the projection centers for the color and depth images is critical for providing geometrically reliable, dense RGB-D data.

Deep Dive: Global Registration

The initial pose estimation for each scan position is completed during the data acquisition phase. Whenever a new location is scanned, the system begins a search for potential pairs among all previously completed scan positions and compute feature matches. By combining the calculated feature matches with the corresponding 3D point cloud data, the relative pose is determined. Once the correct pose is confirmed, the new scan position is registered into the existing map.

Since fully automatic registering occasionally results in errors, manual correction is sometimes applied. The initial poses then undergo a global post-processing step for further refinement. This global post-processing considers the co-visibility relationships between all scan positions, using this framework to perform a global optimization incorporating both visual and 3D features. This process ultimately generates the final, corrected pose file.

Real World HDR RGB Panorama - High Dynamic Range Indoor Image — HDR RGB Panorama

Real World Depth Map - LiDAR-derived Depth Image — Depth Map

Real World Semantic Segmentation - Per-pixel Labels — Segmentation

Synthetic Scene Generation

To achieve the scale required for modern deep learning, we procedurally generated 9,000 scenes based on real-world floorplan distributions. Our pipeline ensures geometric plausibility and photorealism.

Intelligent Layout Algorithms

Unlike common random generation, our approach is grounded in real-world floorplan distributions. We employ a hybrid optimization system integrating placement priors and expert design knowledge.

High-Fidelity Rendering & Assets

We leverage a massive library of over 100,000 high-precision PBR models and a modified Unreal Engine 5 pipeline with hardware-accelerated Ray Tracing.

Deep Dive: Layout Logic

We developed a hybrid rule-based and learning-based discrete space optimization system. This system integrates two critical components: placement priors derived from Realsee's vast real scene data, and expert knowledge provided by professional interior designers (e.g., circulation flow and storage requirements).

This integration ensures that all generated layouts are structurally valid and functionally plausible. Furthermore, we guarantee aesthetic diversity by employing over 200 distinct style templates and a style matching algorithm, supporting a wide range of specialized room types such as children's rooms, gyms, and tea rooms.

Deep Dive: Rendering Pipeline

3D Asset Library

Our Realsee "Weilaijia" model library contains >100,000 models across 200+ categories. Unlike low-poly approximations, our models preserve fine geometric details and use PBR materials for realistic surface properties (specular highlights, subsurface scattering).

Ray-Tracing Engine

Built on a modified Unreal Engine 5, our pipeline utilizes Ray Tracing for accurate Global Illumination (GI), Reflections, and Ambient Occlusion. We integrate DLSS and five distinct illumination schemas including Warm Day, Cold Day, Natural Day, Warm Night, and Cold Night to maximize domain variability.

Synthetic Rendered RGB Panorama - Photorealistic Indoor Scene — Rendered RGB Panorama

Synthetic Depth Map - Ground Truth Depth — Depth Map

Synthetic Surface Normals - Per-pixel Normal Vectors — Surface Normals

Synthetic Semantic Segmentation - Object Instance Labels — Segmentation

Download Dataset

To access the dataset, you must sign a Data Usage Agreement (PDF format). You can download the agreement directly below. Please send your request, specifying your intended use, to developer@realsee.com. Once your application is approved and you have signed the agreement, we will reply with download instructions and links.

Already have Phase I access? You do not need to submit a new agreement. Simply forward your previous approval email to developer@realsee.com, including a Google-Drive-accessible email address, and we will grant you download access to the Phase II data directly.

For the data format, the semantic segmentation segment.png encoding, the class-mapping JSON files (real / synthetic) with the official colour palette, and the visualization script, see the dataset structure documentation.

Download Data Usage Agreement (PDF)

Frequently Asked Questions

What is Realsee3D Dataset?

Realsee3D is a large-scale multi-view RGB-D dataset containing 10,000 complete indoor scenes (1,000 real-world + 9,000 synthetic). It is designed to advance research in indoor 3D perception, reconstruction, and scene understanding, providing high-fidelity panoramic RGB images, depth maps, semantic segmentation labels, and camera poses.

How can I download the Realsee3D Dataset?

To access the dataset, you must sign a Data Usage Agreement (PDF format). You can download the agreement directly here. Please send your request, specifying your intended use, to developer@realsee.com. Once your application is approved and you have signed the agreement, you will receive download instructions and links. If you already have Phase I access, you do not need to submit a new agreement — simply forward your previous approval email (including a Google-Drive-accessible email address) to developer@realsee.com to receive Phase II download access.

What data formats are included in Realsee3D?

Realsee3D provides multiple data modalities including: HDR panoramic RGB images (equirectangular projection), depth maps synthesized from LiDAR point clouds, and per-viewpoint semantic segmentation maps (segment.png, available for both real-world and synthetic scenes — real-world labels are current-model predictions, not manual annotation). Additional modalities such as surface normal maps (synthetic only), CAD drawings, floorplans, and 3D object detection annotations are forthcoming.

What are the licensing terms for Realsee3D Dataset?

The Realsee3D Dataset is available for academic and research purposes under a Data Usage Agreement. Users must apply by contacting developer@realsee.com and agree to the terms before accessing the data. Commercial use requires separate licensing arrangements.

How was the real-world data in Realsee3D collected?

Real-world data was collected using the Realsee Galois P4 3D LiDAR scanning system, which captures high-fidelity synchronized RGB-D data with nearly perfectly co-centered cameras and LiDAR. The system achieves HDR quality via multi-exposure stacking and ensures precise alignment between depth and color data through calibrated camera-LiDAR extrinsics.

Changelog

2026-07-01: Our new work Argus — a metric panoramic 3D reconstruction method trained on Realsee3D (ECCV 2026) — is now released. The covisibility score matrix data used by Argus is now available in Realsee3D.
2026-06-25: Phase II data released — per-viewpoint pixel-wise semantic segmentation (segment.png) for both real-world and synthetic scenes. Real-world maps are current-model predictions (not manual annotation); ground-truth annotations are planned for a future release.
2025-12-05: Phase I data(RGB-D pano and extrinsics) release.
2025-11-28: Dataset introduction and official website release.

Built with Realsee3D

ECCV 2026

Argus: Metric Panoramic 3D Reconstruction for Indoor Scenes is our own work, developed by the same Realsee team behind this dataset. Argus is a data-driven feed-forward network trained on Realsee3D that reconstructs complete, geometrically consistent, metric-scale indoor 3D scenes from sparse, unordered panoramic images in a single forward pass. It achieves state-of-the-art metric performance on camera pose estimation, depth estimation, and point cloud reconstruction on the Realsee3D benchmark, demonstrating the value and usability of the dataset for panoramic 3D reconstruction research.

Project Page Paper Code Model Demo

Citation

If you use the Realsee3D dataset in your research, please cite our paper:

@misc{Li2025realsee3d_data,
  doi = {10.5281/zenodo.17826242},
  url = {https://doi.org/10.5281/zenodo.17826242},
  author = {Li, Linyuan and Wu, Yan and Li, Xi and Wang, Lingli and Rao, Tong and Zhou, Jie and Pan, Cihui and Hui, Xinchen},
  title = {Realsee3D: A Large-Scale Multi-View RGB-D Dataset of Indoor Scenes (Version 1.1)},
  publisher = {Zenodo},
  year = {2026}
}

@misc{li2026argusmetricpanoramic3d,
  title={Argus: Metric Panoramic 3D Reconstruction for Indoor Scenes},
  author={Xi Li and Linyuan Li and Yan Wu and Tong Rao and Kai Zhang and Xinchen Hui and Cihui Pan},
  year={2026},
  eprint={2606.30047},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2606.30047}
}

Collaborate With Us

We welcome academic and industry professionals interested in the Realsee3D dataset to contact us for further cooperation.

Please reach out to us at: pancihui001@realsee.com