I’ve introduced my tracking algorithm in the previous post. One of the issue I have is that the point cloud generated by my SFMToolkit (using Bundler) is not always accurate. This is a list of structure from motion projects alternative I’m interested in:
The idea of this pose estimator is based on PTAM(Parallel Tracking and Mapping). PTAM is capable of tracking in an unknown environment thanks to the mapping done in parallel. But in fact if you want to augment reality, it’s generally because you already know what you are looking at. So, being able to have a tracking working in an unknown environment is not always needed. My idea was simple: instead of doing a mapping in parallel, why not using SFM in a pre-processing step ?
input: point cloud + camera shot
output: position and orientation of the camera
So my outdoor tracking algorithm will eventually work like this:
pre-processing step
generate a point cloud of the outdoor scene you want to track using Bundler
create a binary file with a descriptor (Sift/Surf) per vertex of the point cloud
The tricky part is that absolute pose computation could last several “relative pose” estimation. So once you’ve got the absolute pose you’ll have to compensate the delay by cumulating the previous relative pose…
This is what I’ve got so far:
pre-processing step: binary file generated using SiftGPU (planning to move on my GPUSurf implementation) and Bundler (planning to move on Insight3D or implement it myself using sba)
relative pose: I don’t have an implementation of the relative pose estimator
absolute pose: it’s basically working but needs some improvements:
switch feature extraction/matching from Sift to Surf
remove unused descriptors to speed-up maching step (by scoring descriptors used as inlier with training data)
use another PnP solver (or add ransac to support outliers and have more accurate results)
I have taken a new set of picture of the “Porte Cailhau” in Bordeaux. And I have used one of my tools (BundlerMatcher) to compute image matching using SiftGPU. BundlerMatcher generates a file compatible with Bundler match file. So using BundlerMatcher you can skip the long pre-processing step of feature extraction and image matching and enjoy GPU acceleration!
I have used the “bundle.out” file produced by Bundler to get cameras informations:
intrinsic parameters: focal, distorsion
extrinsic parameters: position, orientation
With these informations you can see the point cloud through the viewpoint of one of the camera registered by Bundler. I’ve added this feature to my current Ogre3D PlyReader. I also have added a background plane to be able to see the picture taken from this viewpoint. This demo is not available for download right now, but you can still watch the video :
The Ogre3D PlyReader and BundlerMatcher will eventually be added to my SVN. I’m currently busy working on another demo, so stay tuned !
In the previous post I’ve been announcing GPU-Surf first release. Now I’m glad to show you a live video demo of GPU-Surf and another demo using Bundler (structure from motion tools):
You’ll get more information on the dedicated demo section.
In this video GPU-Surf was running slowly because of Ogre::Canvas but it should be running really faster.
I have created a very simple PlyReader using Ogre3D, the first version was using billboard to display point cloud but it was slow (30fps with 130k points). Now I’m using custom vertex buffer and it runs at 800fps with 130k points.
The reconstruction was done by the team who created Bundler from 715 pictures of Notre-Dame de Paris (thanks to Flickr). In fact, in this demo they have done the big part of the job, I have just grab their output to check if my PlyReader was capable of reading such a big file.
PlyReader displaying my own dataset
If you already used Bundler you know that structure from motion algorithm needs a very slow pre-processing step to get “matches” between pictures of the dataset. Bundler is packaged to use Lowe’s Sift binary, but it’s very slow because it’s taking pgm as picture input and the output is written in a text file. Then a matching step is executed using KeyMatchFull.exe which is optimized using libANN but still very slow.
I have replaced the feature extraction and matching steps by my own tool: BundlerMatcher. It is using SiftGPU, which gives a very nice speed-up. As my current implementation of GPU-Surf isn’t complete I can’t use it instead of SiftGPU but this is my intention.
23 pictures taken with a classic camera (Canon Powershot A700)
I have created this dataset with my camera and matched the pictures using my own tool: BundlerMatcher. This tool creates the same .key file as Lowe Sift tool and creates a matches.txt file that is used by Bundler. I have tried to get rid off this temporary .key file and keep everything in memory but changing Bundler code to handle this structure was harder than I predicted… I’m now more interested by insight3d implementation (presentation, source) which seems to be easier to hack with.
I'm Henri Astre, a French guy who likes visual experiments. I've done some nice demos in various domains (augmented reality, GPU computing, structure from motion, 3d rendering, web app...) that you can see on this blog. [+ resume]
