Theia：基于 Eigen3 和 Ceres 的 SfM 库，含高质量的算法实现；

MVE：易用的基于图像三维建模的框架。

代码结构

  • Image
  • Features = Keypoints + Descriptors
    • Matching
  • RANSAC
  • Pose and Resectioning(Calibration $\mathbf{P_{3\times 4}}$)
    • P3P
      • A Novel Parameterization of the Perspective-Three-Point Problem for a direct computation of Absolute Camera position and Orientation
    • 5点解E $E=[t]_{\times} * R,\quad y^\top * E * x = 0$
      • Recent Developments on Direct Relative Orientation
    • 4点解H
      • DLT@MVG
    • 8点解F $x’ F x = 0$
      • @MVG
    • PnP(Perspective N-Point)
      • A Direct Least-Squares (DLS) Method for PnP
    • Four Point Focal Length
    • Five Point Focal Length and Radial Distortion
    • Three Point Relative Pose with a Partially Known Rotation
    • Four Point Relative Pose with a Partially Known Rotation
    • Two Point Absolute Pose with a Partially Known Rotation
  • Math
    • N次多项式解析解（N<4）
    • N次多项式迭代解
    • Sequential Probability Ratio Test（SRPT）？
  • Camera Models
    • Pinhole Camera Model \(K = \begin{bmatrix}f & s & p_x \\ 0 & f * a & p_y \\ 0 & 0 & 1 \end{bmatrix}\)
    • Fisheye Camera Model \(r = \sqrt{x^2 + y^2} \theta = atan2(r, |z|) \\ \theta_d = \theta (1 + k1 *\theta^2 + k2* \theta^4 + k3 *\theta^6 + k4* \theta^8) \\ x' = \theta_d *x / r \\ y' = \theta_d* y / r\)
  • SfM
    • Pipeline
      • 特征提取
      • 特征匹配
      • 位姿估计
        • Estimate camera poses from two-view matches and geometries using incremental or global SfM
      • 三角化
      • BA优化（调用 Ceres）
    • 基本数据结构
      • View
      • Track
      • Reconstruction = Views + Tracks
      • ViewGraph = View(Vertex) + TwoViewInfo(Edge)

image-20190804152139289

Incremental SfM Pipeline

The incremental SfM pipeline is as follows:

  1. Choose an initial camera pair to reconstruct.
  2. Estimate 3D structure of the scene.
  3. Bundle adjustment on the 2-view reconstruction.
  4. Localize a new camera to the current 3D points. Choose the camera that observes the most 3D points currently in the scene.
  5. Estimate new 3D structure.
  6. Bundle adjustment if the model has grown by more than 5% since the last bundle adjustment.
  7. Repeat steps 4-6 until all cameras have been added.

Global SfM Pipeline

  1. Create the initial view graph from 2-view matches and TwoViewInfo that describes the relative pose between matched images.
  2. Filter initial view graph and remove outlier 2-view matches.
  3. Calibrate internal parameters of all cameras (either from EXIF or another calibration method).
  4. Estimate global orientations of each camera with a RotationEstimator
  5. Filter the view graph: remove any TwoViewInfos where the relative rotation does not agree with the estimated global rotations.
  6. Refine the relative translation estimation to account for the estimated global rotations.
  7. Filter any bad TwoViewInfo based on the relative translations.
  8. Estimate the global positions of all cameras from the estimated rotations and TwoViewInfousing a PositionEstimator
  9. Estimate 3D points.
  10. Bundle adjust the reconstruction.
  11. (Optional) Attempt to estimate any remaining 3D points and bundle adjust again.

三维重建

参考官网说明安装TheiaSfM和MVE，MVE带三维重建的整个流程，MVE的依赖项少，编译安装相对轻松，核心操作流程包括：

  1. Creating a dataset, by converting input photos into the MVE File Format.
  2. Structure from Motion, which reconstructs the camera parameters.
  3. Multi-View Stereo, which reconstructs dense depth maps for each photo.
  4. Surface Reconstruction, which reconstructs a surface mesh.

对应程序执行如下

makescene -i <image-dir> <scene-dir>
sfmrecon <scene-dir>
dmrecon -s2 <scene-dir>
scene2pset -F2 <scene-dir> <scene-dir>/pset-L2.ply
fssrecon <scene-dir>/pset-L2.ply <scene-dir>/surface-L2.ply
meshclean -t10 <scene-dir>/surface-L2.ply <scene-dir>/surface-L2-clean.ply

实践中发现，MVE的SfM执行最耗时间，最近刚好又在学习Theia，故想到用Theia代替MVE的SfM：

  1. 先用convert_theia_reconstruction_to_bundler_file.cc把Theia的SfM结果进行转换Bundler的格式；
  2. 用makescene -bundle-id=0 ......生成MVE格式；
  3. 最后安装MVE流程执行稠密重建MVS生成若干深度图，深度图融合生成稠密点云，表面重建。

NB：执行dmrecon前需要修改list.txt中的图像名，否则无法生成深度图！

实验结果

用iPhone SE采集了19张图像，用TheiaSfM代替MVE自带的SfM可以节省30%的时间，重建结果如下：

snapshot00

细心的你通过观察发现，弱纹理的区域很难重建，会出现空洞。

参考

Awesome 3D reconstruction list