Building Rome in a Day:

Project home is using Bundler (GPL)

Building Rome on a Cloudless Day:

Project home | Source code (Non-profit license, I’ve ported their source to windows)

Samantha:

Project home (I’ve contacted them without response but they said that they were going to release the source code: check at 28:50)

Samantha	Bundler

PhotoSynth:

Website – Microsoft closed-source SFM application: check out my PhotoSynthToolkit

PhotoSynth	Bundler

ETH-V3D Structure-and-Motion software:

Project home with source code (GPL, I’ve partially ported it to windows)

Simple Sparse Bundle Adjustment:

Project home with source code (LGPL, I’ve ported it to windows)

A multi-stage linear approach to structure from motion:

Project home | paper

Results from the paper of LinearSFM (Microsoft Research)

This list is not exhaustive, I’ve seen other projects (Efficient Large Scale Multi-View Stereo for Ultra High Resolution Image Sets: not sure how it is related to ETH-V3D Structure-and-Motion software)

input: point cloud + camera shot	output: position and orientation of the camera

So my outdoor tracking algorithm will eventually work like this:

• pre-processing step
  • generate a point cloud of the outdoor scene you want to track using Bundler
  • create a binary file with a descriptor (Sift/Surf) per vertex of the point cloud
• in real-time, for each frame N:
  • extract feature using FAST
  • match feature from frame N-1 using 2D patch
  • compute “relative pose” between frame N and N-1
• in almost real-time, for each “key frame”:
  • extract feature and descriptor
  • match descriptor with those of the point cloud
  • generate 2D/3D correspondence from matches
  • compute “absolute pose” using PnP solver (EPnP)

The tricky part is that absolute pose computation could last several “relative pose” estimation. So once you’ve got the absolute pose you’ll have to compensate the delay by cumulating the previous relative pose…

This is what I’ve got so far:

• pre-processing step: binary file generated using SiftGPU (planning to move on my GPUSurf implementation) and Bundler (planning to move on Insight3D or implement it myself using sba)
• relative pose: I don’t have an implementation of the relative pose estimator
• absolute pose: it’s basically working but needs some improvements:
  • switch feature extraction/matching from Sift to Surf
  • remove unused descriptors to speed-up maching step (by scoring descriptors used as inlier with training data)
  • use another PnP solver (or add ransac to support outliers and have more accurate results)

I have used the “bundle.out” file produced by Bundler to get cameras informations:

• intrinsic parameters: focal, distorsion
• extrinsic parameters: position, orientation

With these informations you can see the point cloud through the viewpoint of one of the camera registered by Bundler. I’ve added this feature to my current Ogre3D PlyReader. I also have added a background plane to be able to see the picture taken from this viewpoint. This demo is not available for download right now, but you can still watch the video :

The Ogre3D PlyReader and BundlerMatcher will eventually be added to my SVN. I’m currently busy working on another demo, so stay tuned !

There are three demos in this video:

1. GPU-Surf live demo.
2. PlyReader displaying Notre-Dame dataset.
3. PlyReader displaying my own dataset (Place de la Bourse, Bordeaux).

GPU-Surf

You’ll get more information on the dedicated demo section.
In this video GPU-Surf was running slowly because of Ogre::Canvas but it should be running really faster.

PlyReader displaying Notre-Dame dataset

I’m also interested in structure from motion algorithm, that’s why I have tested Bundler, which comes with a good dataset of Notre-Dame de Paris.

I have created a very simple PlyReader using Ogre3D, the first version was using billboard to display point cloud but it was slow (30fps with 130k points). Now I’m using custom vertex buffer and it runs at 800fps with 130k points.

The reconstruction was done by the team who created Bundler from 715 pictures of Notre-Dame de Paris (thanks to Flickr). In fact, in this demo they have done the big part of the job, I have just grab their output to check if my PlyReader was capable of reading such a big file.

PlyReader displaying my own dataset

If you already used Bundler you know that structure from motion algorithm needs a very slow pre-processing step to get “matches” between pictures of the dataset. Bundler is packaged to use Lowe’s Sift binary, but it’s very slow because it’s taking pgm as picture input and the output is written in a text file. Then a matching step is executed using KeyMatchFull.exe which is optimized using libANN but still very slow.

I have replaced the feature extraction and matching steps by my own tool: BundlerMatcher. It is using SiftGPU, which gives a very nice speed-up. As my current implementation of GPU-Surf isn’t complete I can’t use it instead of SiftGPU but this is my intention.

23 pictures taken with a classic camera (Canon Powershot A700)	Point cloud generated using Bundler

I have created this dataset with my camera and matched the pictures using my own tool: BundlerMatcher. This tool creates the same .key file as Lowe Sift tool and creates a matches.txt file that is used by Bundler. I have tried to get rid off this temporary .key file and keep everything in memory but changing Bundler code to handle this structure was harder than I predicted… I’m now more interested by insight3d implementation (presentation, source) which seems to be easier to hack with.