Skip to content

3DCV Workshop 04 - 3D Scene Reconstruction

Workshop Overview

In this workshop, we'll reconstruct a room or a larger outdoor scene and compare various methods and devices to realize this task. The goal is to understand the strengths and weaknesses of different approaches, including traditional Structure From Motion methods and model-based methods.

Workshop Goals

  • Reconstruct a room or a larger outdoor scene using various methods.
  • Compare results from different devices and techniques.
  • Reflect on the effectiveness of traditional versus model-based methods.

Procedure

The parts do not need to be done by everyone. Form teams and pairs that work on each part and present the status and results every hour.

Parts to do:

  • Collect images, use a smartphone or a better camera.
  • Reconstruct the object using one of the following methods:
    • Structure From Motion: Use software like Meshroom or COLMAP to create a 3D model of the scene from images.
    • Model-based Reconstruction: Explore visual geometric models like VGGT, DUNE or Mast3r that can reconstruct 3D scenes from images.

Expected Outcomes

  • A reconstructed 3D model of a room or a larger outdoor scene, exported in a common format (e.g., OBJ, PLY).
  • A comparison of the results from different methods and devices.
  • A reflection on the effectiveness of traditional versus model-based methods.
  • A written report summarizing the findings.
  • A discussion on the future of 3D scene reconstruction in the context of AI advancements.
  • A reproducible workflow that can be shared with others.

3D Reconstruction from Real-World Image Data

Capturing

Capture RGB images of some room or outdoor scene that you want to reconstruct. Use a smartphone or or a better camera.

Hints for capturing

Work like a scientist. Think before you act:

  • Which properties should the scene have, so that the reconstruction is easy?
  • Capture scenes with good lighting conditions. Consider lens distortion and reflections.
  • Note for all your samples your expectations. Name and sort all captured scenes and images in a folder structure that allows you to automatically process all images using a script.

Image preprocessing

Inspect the captured images and prepare them for reconstruction. If necessary, use a tool like SAM to segment the scene or create image masks.

Reconstructing the 3D scene

Pick one of the following tools to reconstruct the 3D scene from the captured images:

Hints for working with tools

Work like a scientist. Think before you act:

  • Check the documentation of the tool you selected.
  • Compare the system requirements of the tools with your computer.
  • Start with the example images provided by the tool to ensure it works correctly. If not available, use synthetic images from the ETH3D dataset (available on the mobile hard drive) or similar.

3D Reconstruction with AI Models

Pick one or more of the following models to run 3D reconstruction on RGB images:

You can also check out more models from this list of Awesome DUSt3R Resources

Hints for working with scientific tools

Work like a scientist. Think before you act:

  • Check the documentation (GitHub readme) of the tool you selected.
  • Compare the system requirements of the tools with your computer. Usually this is not stated prominently, so you may need to check the code or issues. And be very precise about the hard- and software (driver versions, CUDA version) you have available.
  • Start with the example images provided by the tool to ensure it works correctly. If not available, use synthetic images from the ETH3D dataset (available on the mobile hard drive) or similar.

Discussion and Reflection

Key Question

Do we still need COLMAP (feature detection) or are the AI models good enough?

Points to Consider

  • Accuracy and consistency of AI models across different examples
  • Computational requirements of AI models
  • Limitations in specific use cases
  • Real-time capabilities

Task

Create the following deliverables and upload them to the FELIX folder:

  • A reconstructed 3D model of a room or outdoor scene, exported in a common format (e.g., OBJ, PLY).
  • A comparison of the results from different methods and devices.
  • A reflection on the effectiveness of traditional feature-based versus model-based methods.
  • A written report summarizing the findings.
  • A discussion on the future of 3D scene reconstruction in the context of AI advancements.
  • A reproducible workflow that can be shared with others.

Final Thoughts

By completing this workshop, you have:

  • Captured RGB data (aka images) for a small research project,
  • Organized and structured data to efficiently process it,
  • Dealt with classic photogrammetry tools as well as scientific code that uses AI models,
  • and presented and discussed your results in a scientific format.

Grading

For this workshop and the last one the outcome of task needs to be handed in. Upload the resulting document to the FELIX folder until one week after the workshops. It should either contain the files in a zip or a link to the project repository. If you worked in teams, state who is responsible for each part.

Your results need to be reproducible and contain references to all used sources and tools.