This viewer is now integrated with the new version of my PhotoSynthToolkit (v5). This toolkit allow you to download synth point cloud and thumbnails pictures. You can also densify the sparse point cloud generated by PhotoSynth using PMVS2 and then create great accurate mesh using MeshLab.

New feature of PhotoSynthToolkit v5:

• Thumbnails downloading should be faster (8x)
• New C++ HD picture downloader (download tiles and re-compose them)
• Tools to generate “vis.dat” from previous PMVS2 call (analysing .patch file)
• Working Ogre3D PhotoSynth viewer:
  • Can read dense point cloud created with my PhotoSynthToolkit using PMVS2
  • Click on a picture to change camera viewpoint
  • No-roll camera system

Warning: the PhotoSynth viewer may need a very powerful GPU (depending on the synth complexity: point cloud size and number of thumbnails). I’ve currently tested a scene with 820 pictures and 900k vertices on a Nvidia 8800 GTX with 768mo and it was working at 25fps (75fps with a 470 GTX and 1280mo). I wish I could have used Microsoft Seadragon .

Download:

The PhotoSynthToolkit v5 is available on his dedicated page, please do not make direct link to the zip file but to this page instead. So people willing to download the toolkit will always get the latest version.

Video demo:

Future version

Josh Harle has created CameraExport: a solution for 3DS Max that enable to render the picture of the Synth using camera projection. I don’t have tested it yet but I’ll try to generate a file compatible with his 3DS Max script directly from my toolkit, thus avoiding to download the Synth again using a modified version of SynthExport. Josh has also created a very interesting tutorial on how to use mask with PMVS2:

Masks with the PhotoSynth Toolkit 4 – tutorial from Josh Harle on Vimeo.

2010 was a year full of visual experiments for me, I hope that you like what you see on this blog. In this post I’m making a little overview of all visual experiments created by me during this year. This is an opportunity to catch-up something you’ve missed! I’d like also to thanks some person that have been helping me too:

• Olafur Haraldsson: for creating the photogrammetry forum
• Josh Harle: for his videos tutorials and his nice blog
• You: for reading this

Visual experiments created in 2010:

During this year I have added some features to Ogre3D:

• ArToolKitPlus: augmented reality marker-based system
• Cuda: for beginner only (at least advanced user could grab some useful code)
• OpenCL: for beginner only (at least advanced user could grab some useful code)
• Html5 Canvas: implementation based on skia for graphics and V8 for javascript scripting
• Kinect: this is a very hacky solution, I’ll improve it later

I also have learned GPGPU programming by myself while coding a partial GPUSurf implementation based on Nico Cornelis paper. But this implementation is not complete and I’m willing to rewrite it with a GPGPU framework based on OpenGL and CG only (not Ogre3D). With such a framework writing Sift/Surf detector should be easier and more efficient.

I have created some visual experiments related to Augmented Reality:

• Remote AR prototype
• Outdoor 3D tracking using point cloud generated by structure from motion software
• Outdoor 2D tracking using panorama image

My outdoor 3D tracking algorithm for augmented reality needs an accurate point cloud: this is why I’m interested in structure from motion and I’ve created two SfM toolkit:

• SFMToolkit (SiftGPU -> Bundler -> CMVS -> PMVS2)
• PhotoSynthToolkit (PhotoSynth -> PMVS2)

Posts published in 2010:

• 2010/12/22: Outdoor tracking using panoramic image
• 2010/12/20: Structure from motion projects
• 2010/12/13: Augmented Reality outdoor tracking becoming reality
• 2010/11/20: Kinect experiment with Ogre3D
• 2010/11/19: PhotoSynthToolkit results
• 2010/11/09: PhotoSynth Toolkit updated
• 2010/11/05: Structure From Motion Toolkit released
• 2010/09/27: My 5 years old Quiksee competitor
• 2010/09/23: PMVS2 x64 and videos tutorials
• 2010/09/08: Introducing OpenSynther
• 2010/08/22: Dense point cloud created with PhotoSynth and PMVS2
• 2010/08/19: My PhotoSynth ToolKit
• 2010/07/12: Pose Estimation using SfM point cloud
• 2010/07/11: Remote Augmented Reality Prototype
• 2010/07/08: Structure From Motion Experiment
• 2010/06/25: GPU-Surf video demo
• 2010/06/23: GPUSurf and Ogre::GPGPU
• 2010/06/20: Ogre::Canvas, a 2D API for Ogre3D
• 2010/05/09: Ogre::OpenCL and Ogre::Canvas
• 2010/04/26: Cuda integration with Ogre3D
• 2010/04/09: Multitouch prototype done using Awesomium and Ogre3D
• 2010/03/05: ArToolKitPlus integration with Ogre3D
• 2010/02/20: Hello World !

First of all, I must admit that this is more a “proof-of-concept” rather than a prototype… But the goal was to illustrate a concept needed for my job. I love this kind of challenge! Building something like this in 2 days was only possible thanks to great open-source library:

• Ogre3D (MIT)
• OpenCV (BSD)
• SiftGPU (non-profit license)

Analysis

I’m using a panoramic image as reference. For each frame of the video I’m extracting Sift feature using SiftGPU and matching them with those of the reference image. Then I’m computing the homography between the 2 images using Ransac homography estimator (OpenCV cvFindHomography).

Performance

The performance are low due to complexity of the Sift detection and matching and that I’m applying the homography using cvWarpPerspective.

I’m working on another version using Fast (or Agast) as feature detector and Brief as descriptor. This should lead to a significant speed-up and may eventually run on a mobile… Using the GPU vertex and pixel shader instead of the CPU to apply the homography should also gives a nice speed-up.

I’m also aware that it is not correct to apply an homography on a cylindric panoramic image (especially if you don’t undistort the input video frame too )

Analysis

In a pre-processing step I’ve built a sparse point cloud of the place using my SFMToolkit. Each vertex of the point cloud has several 2D Sift features correspondences. I’ve only kept one Sift descriptor per vertex (mean of the descriptors) and put all descriptors in an index using Flann.

For each frame of the video to be augmented, I’ve extracted Sift feature with SiftGPU and then matched them using Flann 2-nearest neighbor search and a distance ratio threshold. The Flann matching is done in parallel with boost::threadpool. The matches computed contains a lot of outliers. So I have implemented a Ransac pose estimator using EPnP that permits to filter bad 2d/3d correspondences.

Performance

My implementation is slow (due to my implementation of Ransac EPnP that could be improved).

The slowness is not a so big issue because it doesn’t need to run at 30fps. Indeed the goal of my prototype is to have absolute pose with this tracking system each second and relative pose using inertial system available on mobile device (or using KLT tracking).

Issue

• Performance (faster is better )
• Point cloud reference is not always accurate (Bundler fault)

In another post I’ll introduce alternative to Bundler: faster and more accurate.

This demo is a ripp-off the Kinect-v11 demo made by Zephod. In fact I’ve designed a new Ogre::Kinect library that provide Kinect connection through Zephod library. Then I’ve replace the Zephod OpenGL demo by an Ogre3D demo using my library. The nice part is that I’ve managed to move some depth to rgb conversion to the GPU (using pixel shader).

Links

Binary demo: OgreKinectDemo1.zip
Source code: svn on code.google
Documentation: doxygen
License: MIT

Download location and source code introduced in my previous post.

I have updated my PhotoSynth toolkit for easier usage (the same way as SFMToolkit). This is an example of dense mesh creation from 12 pictures using this toolkit:

The 12 pictures were shot with a Canon PowerShot A700:

Thanks to this toolkit, PMVS2 and MeshLab you can create a dense mesh from these 12 pictures:

triangulated mesh with vertex color -> triangulated mesh with vertex color and SSAO -> triangulated mesh shaded with SSAO -> triangulated mesh wireframe -> photosynth sparse point cloud
(sparse point cloud : 8600 vertices, dense point cloud: 417k vertices, mesh: 917k triangles)

You can also take a loot at the PhotoSynth reconstruction of the sculpture.

PhotoSynthToolkit is composed of several programs:

• PhotoSynthDownloader: download PhotoSynth point cloud and cameras parameters
• PhotoSynth2PMVS: enable to run PMVS2 with a downloaded PhotoSynth point cloud
• PMVS2 : http://grail.cs.washington.edu/software/pmvs/ created by Yasutaka Furukawa
• PhotoSynthViewer: Ogre3D PhotoSynth viewer [not working yet]

Download

The source code is available under MIT license on my github. I have also released a win32 binary version with windows scripting (WSH) for easier usage: PhotoSynthToolkit4.zip.

Help

If you need some help or just want to discuss about photogrammetry, please join the photogrammetry forum created by Olafur Haraldsson. You may also be interested by Josh Harle’s video tutorials, they are partially out-dated due to the new PhotoSynthToolkit version but these videos are very good to learn how to use MeshLab.

Please go to the PhotoSynthToolkit page to get the latest version

I have finally released my Structure-From-Motion Toolkit (SFMToolkit). So what can you do with it ? Let’s say you have a nice place like the one just bellow:

Place de la Bourse, Bordeaux, FRANCE (picture from Bing)

Well, now you can take a lot of pictures of the place (around 50 in my case):

And then compute structure from motion and get a sparse point cloud using Bundler:

Finally you have a dense point cloud divided in cluster by CMVS and computed by PMVS2:

You can also take a loot at the PhotoSynth reconstruction of the place with 53 pictures and 26 (without the fountain).

This is the SFMToolkit workflow:

SFMToolkit is composed of several programs:

• BundlerFocalExtractor : extract CCD width from Exif using XML database.
• BundlerMatcher : extract and match feature using SiftGPU.
• Bundler : http://phototour.cs.washington.edu/bundler/ created by Noah Snavely.
• CMVS : http://grail.cs.washington.edu/software/cmvs/ created by Yasutaka Furukawa.
• PMVS2 : http://grail.cs.washington.edu/software/pmvs/ created by Yasutaka Furukawa.
• BundlerViewer : Bundler and PMVS2 output viewer based on Ogre3D (OpenSource 3D rendering engine).

Download

As you can see this “toolkit” is composed of several open-source component. This is why I have decided to open-source my part of the job too. You can download the source code from the SFMToolkit github. You can also download a pre-compiled x64 version of the toolkit with windows scripting (WSH) for easier usage (but not cross-platform): SFMToolkit1.zip.

Help

If you need some help or just want to discuss about photogrammetry, please join the photogrammetry forum created by Olafur Haraldsson. You may also be interested by Josh Harle’s video tutorials, they are partially out-dated due to the new SFMToolkit but these videos are very good to learn how to use MeshLab.

Please go to the SFMToolkit page to get the latest version

PhotoSynth sparse point-cloud 11k vertices	PMVS2 dense point-cloud 230k vertices

I also have created a web app : PhotoSynthTileDownloader that permit to download all pictures of a synth in HD. I didn’t have release it yet because I’m concerned about the legal issue, but you can see that it’s already working by yourself:

I’ll give more information about it in a few day, stay tuned !

Edit: I have removed the worflow graph and moved it on my next post.

Please go to the PhotoSynthToolkit page to get the latest version

input: point cloud + camera shot	output: position and orientation of the camera

So my outdoor tracking algorithm will eventually work like this:

• pre-processing step
  • generate a point cloud of the outdoor scene you want to track using Bundler
  • create a binary file with a descriptor (Sift/Surf) per vertex of the point cloud
• in real-time, for each frame N:
  • extract feature using FAST
  • match feature from frame N-1 using 2D patch
  • compute “relative pose” between frame N and N-1
• in almost real-time, for each “key frame”:
  • extract feature and descriptor
  • match descriptor with those of the point cloud
  • generate 2D/3D correspondence from matches
  • compute “absolute pose” using PnP solver (EPnP)

The tricky part is that absolute pose computation could last several “relative pose” estimation. So once you’ve got the absolute pose you’ll have to compensate the delay by cumulating the previous relative pose…

This is what I’ve got so far:

• pre-processing step: binary file generated using SiftGPU (planning to move on my GPUSurf implementation) and Bundler (planning to move on Insight3D or implement it myself using sba)
• relative pose: I don’t have an implementation of the relative pose estimator
• absolute pose: it’s basically working but needs some improvements:
  • switch feature extraction/matching from Sift to Surf
  • remove unused descriptors to speed-up maching step (by scoring descriptors used as inlier with training data)
  • use another PnP solver (or add ransac to support outliers and have more accurate results)