I’ve worked a lot on OpenSynther lately: OpenSynther is the name of my structure-from-motion solution. This new version is a major rewrite of the previous version which was using Surf with both GPU and multi-core CPU matching. The new version is using SiftGPU and Flann to achieve linear matching complexity of unstructured input as described in Samantha paper. You can find more information about OpenSynther features on it dedicated page (including source code).
OpenSynther has been designed as a library (OpenSyntherLib) which has already proven to be useful for several programs written by myself:
OpenSynther: work in progress… used by my augmented reality demo
BundlerMatcher: this is the matching solution used by SFMToolkit
Outdoor augmented reality demo using OpenSynther
I’ve improved my first attempt of outdoor augmented reality: I’m now relying on PhotoSynth capability of creating a point cloud of the scene instead of Bundler. Then I’m doing some processing with OpenSynther and here is what what you get:
You can also take a look at the 3 others youtube videos showing this tracking in action around this church: MVI_6380.avi, MVI_6381.avi, MVI_6382.avi.
PhotoSynth2CMVS
This is not ready yet, I still have some stuff to fix before releasing it. But I’m already producing a valid “bundle.out” file compatible with CMVS processing from PhotoSynth. I’ve processed the V3D dataset with PhotoSynth2CMVS and sent the bundle.out file to Olafur Haraldsson who has managed to create the corresponding 36 million vertices point cloud using CMVS and PMVS2:
The new unstructured linear matching is really fast as you can see on the above chart compared to PhotoSynth. But the quality of the generated point cloud is not as good as PhotoSynth.
This benchmark was computed on a Core i7 with an Nvidia 470 GTX. I’ve also compared the quality of the matching methods implemented in OpenSynther (linear VS quadratic). I’ve used Bundler as a comparator with a dataset of 245 pictures:
Linear
Quadratic
Nb pictures registered
193
243
Time spent to register 193 pictures
33min
1h43min
On the one hand, both the matching and the bundle adjustment are faster with linear matching but on the other hand, having only 193 out of 245 pictures registered is not acceptable. I have some idea on how to improve the linear matching pictures registering ratio but this is not implemented yet (this is why PhotoSynth2CMVS is not released for now).
Future
I’ve been playing with LDAHash last week and I’d like to support this in OpenSynther to improve matching speed and accuracy. It would also help to reduce the memory used by OpenSynther (by a factor 16: 128 floats -> 256bits per feature). I’m also wondering if the Cuda knn implementation could speed-up the matching (if applicable)? I ‘d also like to restore the previous Surf version of OpenSynther which was really fun to implement. Adding a sequential bundle adjustment (as in bundler) would be really interesting too…
Off-topic
I’ve made some modifications to my blog: switched to WordPress 3.x, activated page caching, added social sharing buttons and added my LinkedIn account next to the donate button…
First of all, I’m sorry this post is not about CMVS support in PhotoSynthToolkit . Releasing the PhotoSynthToolkit with CMVS support is way more complicated than predicted… This is because it is not just a file conversion process (as in my PhotoSynth2PMVS). I have designed a library (OpenSyntherLib) that extract features, match them, build tracks and then triangulate them using PhotoSynth cameras parameters. The problem is that this library is highly configurable to match each dataset needs. So providing an automatic solution with good parameters is difficult.
The new PhotoSynth2CMVS tool generates bundler-compatible file (“bundle.out”). I’ve sent the bundle.out file of the V3D dataset to Olafur Haraldsson and he has managed to create a 36 million vertices point cloud with it! It will be showcased in a next post.
PhotoSynth WebGL Viewer:
As soon as I’ve seen the SpiderGL presentation at the 3D-Arch Workshop I’ve decided to implement a PhotoSynth viewer with WebGL! Thanks to Cesar Andres Lopez I’ve found XB PointStream which is very well designed and documented. Implementing a PhotoSynth Parser turns out to be super easy! Porting my PhotoSynthParser.cpp took less than 10min thanks to the jDataView and binary ajax developed by Vjeux. After 2 hours of hacking I got this first viewer:
I can’t host the viewer on my website has it is using a proxy to fix ajax cross-domain security issue (thus it will kill my bandwidth limit). So I’ve made a video to show you how it looks:
The viewer source code will be hosted on my GitHub and I’m working on a Google Chrome extension to by-pass the need of a proxy… Having a Google Chrome extension that replace the Silverlight viewer on photosynth.net would be nice too (if you don’t have Silverlight).
The 3D-Arch workshop was really inspiring: it was really great meeting people working on the same subject, exchanging idea… and the place was really nice too. I couldn’t resist to create some PhotoSynth of the place for future reconstruction: dragon, eagle, statue, door, …
Andrea Fusiello showcased amazing results with Samantha: reconstruction without any camera calibration needed (neither Exif data).
Marc Pierrot-Deseilligny presented Apero: an open-source bundle adjustment software for automatic calibration and orientation of set of images (needs calibrated camera).
I should have published this post sooner but I wanted to make a double post with my new OpenSynther results… CMVS support in PhotoSynthToolkit is coming! You should expect another post next week with nice results
I’ve tested this toolkit with the dataset given by Christopher Zach (see above screenshot) the reconstruction looks good but I only managed to get partial reconstruction from my own dataset.
MissStereo
Quasi-Euclidean Epipolar Rectification: MissStereo created by Pascal Monasse, Neus Sabater, Zhongwei Tang. The original source code is available at the IPOL related page under GPL license. You can download my windows port as both binary (MissStereo1.zip) and source (github).
I’m interested by this method to estimate the fundamental matrix without prior focal length knowledge.
PhotoSynthToolkit with XSI support
With the help of Julien Carmagnac (3D Graphist and XSI advanced user), I’ve duplicated the 3DS Max texture projection rendering solution for Softimage XSI:
As usual, the new version of PhotoSynthToolkit including this new feature is available on his dedicated page.
SFMToolkit with sequence matching
SFMToolkit was packaged with BundlerMatcher, my own matching solution using SiftGPU: this is a trivial quadratic exhaustive matching implementation. This implementation is well-suited for unstructured (unordered) pictures but if you are using a sequence of images as input (movie) you can reduce the complexity of the matching to linear. You can do that by only comparing frame N with frame N+1, N+2 [...], N+p. I’ve also fixed a bug that may occur on some system (Windows 7 x64 with UAC activated): thanks to Josh Harle for the bug reporting!
The new version of SFMToolkit is available on his dedicated page: this new feature is hidden, you need to edit “1 – Bundler.wsf” and set SEQUENCE_MATCHING_ENABLED to true (replace false by true).
3D-Arch’2011
I’m going to the 3D-Arch’2011 Workshop at Trento: 3D Virtual Reconstruction and Visualization of Complex Architectures. I hope to see amazing things about 3D reconstruction
I'm Henri Astre, a French guy who likes visual experiments. I've done some nice demos in various domains (augmented reality, GPU computing, structure from motion, 3d rendering, web app...) that you can see on this blog. [+ resume]
. Releasing the PhotoSynthToolkit with CMVS support is way more complicated than predicted… This is because it is not just a file conversion process (as in my PhotoSynth2PMVS). I have designed a library (OpenSyntherLib) that extract features, match them, build tracks and then triangulate them using PhotoSynth cameras parameters. The problem is that this library is highly configurable to match each dataset needs. So providing an automatic solution with good parameters is difficult.
