This viewer is now integrated with the new version of my PhotoSynthToolkit (v5). This toolkit allow you to download synth point cloud and thumbnails pictures. You can also densify the sparse point cloud generated by PhotoSynth using PMVS2 and then create great accurate mesh using MeshLab.

New feature of PhotoSynthToolkit v5:

• Thumbnails downloading should be faster (8x)
• New C++ HD picture downloader (download tiles and re-compose them)
• Tools to generate “vis.dat” from previous PMVS2 call (analysing .patch file)
• Working Ogre3D PhotoSynth viewer:
  • Can read dense point cloud created with my PhotoSynthToolkit using PMVS2
  • Click on a picture to change camera viewpoint
  • No-roll camera system

Warning: the PhotoSynth viewer may need a very powerful GPU (depending on the synth complexity: point cloud size and number of thumbnails). I’ve currently tested a scene with 820 pictures and 900k vertices on a Nvidia 8800 GTX with 768mo and it was working at 25fps (75fps with a 470 GTX and 1280mo). I wish I could have used Microsoft Seadragon .

Download:

The PhotoSynthToolkit v5 is available on his dedicated page, please do not make direct link to the zip file but to this page instead. So people willing to download the toolkit will always get the latest version.

Video demo:

Future version

Josh Harle has created CameraExport: a solution for 3DS Max that enable to render the picture of the Synth using camera projection. I don’t have tested it yet but I’ll try to generate a file compatible with his 3DS Max script directly from my toolkit, thus avoiding to download the Synth again using a modified version of SynthExport. Josh has also created a very interesting tutorial on how to use mask with PMVS2:

Masks with the PhotoSynth Toolkit 4 – tutorial from Josh Harle on Vimeo.

2010 was a year full of visual experiments for me, I hope that you like what you see on this blog. In this post I’m making a little overview of all visual experiments created by me during this year. This is an opportunity to catch-up something you’ve missed! I’d like also to thanks some person that have been helping me too:

• Olafur Haraldsson: for creating the photogrammetry forum
• Josh Harle: for his videos tutorials and his nice blog
• You: for reading this

Visual experiments created in 2010:

During this year I have added some features to Ogre3D:

• ArToolKitPlus: augmented reality marker-based system
• Cuda: for beginner only (at least advanced user could grab some useful code)
• OpenCL: for beginner only (at least advanced user could grab some useful code)
• Html5 Canvas: implementation based on skia for graphics and V8 for javascript scripting
• Kinect: this is a very hacky solution, I’ll improve it later

I also have learned GPGPU programming by myself while coding a partial GPUSurf implementation based on Nico Cornelis paper. But this implementation is not complete and I’m willing to rewrite it with a GPGPU framework based on OpenGL and CG only (not Ogre3D). With such a framework writing Sift/Surf detector should be easier and more efficient.

I have created some visual experiments related to Augmented Reality:

• Remote AR prototype
• Outdoor 3D tracking using point cloud generated by structure from motion software
• Outdoor 2D tracking using panorama image

My outdoor 3D tracking algorithm for augmented reality needs an accurate point cloud: this is why I’m interested in structure from motion and I’ve created two SfM toolkit:

• SFMToolkit (SiftGPU -> Bundler -> CMVS -> PMVS2)
• PhotoSynthToolkit (PhotoSynth -> PMVS2)

Posts published in 2010:

• 2010/12/22: Outdoor tracking using panoramic image
• 2010/12/20: Structure from motion projects
• 2010/12/13: Augmented Reality outdoor tracking becoming reality
• 2010/11/20: Kinect experiment with Ogre3D
• 2010/11/19: PhotoSynthToolkit results
• 2010/11/09: PhotoSynth Toolkit updated
• 2010/11/05: Structure From Motion Toolkit released
• 2010/09/27: My 5 years old Quiksee competitor
• 2010/09/23: PMVS2 x64 and videos tutorials
• 2010/09/08: Introducing OpenSynther
• 2010/08/22: Dense point cloud created with PhotoSynth and PMVS2
• 2010/08/19: My PhotoSynth ToolKit
• 2010/07/12: Pose Estimation using SfM point cloud
• 2010/07/11: Remote Augmented Reality Prototype
• 2010/07/08: Structure From Motion Experiment
• 2010/06/25: GPU-Surf video demo
• 2010/06/23: GPUSurf and Ogre::GPGPU
• 2010/06/20: Ogre::Canvas, a 2D API for Ogre3D
• 2010/05/09: Ogre::OpenCL and Ogre::Canvas
• 2010/04/26: Cuda integration with Ogre3D
• 2010/04/09: Multitouch prototype done using Awesomium and Ogre3D
• 2010/03/05: ArToolKitPlus integration with Ogre3D
• 2010/02/20: Hello World !

First of all, I must admit that this is more a “proof-of-concept” rather than a prototype… But the goal was to illustrate a concept needed for my job. I love this kind of challenge! Building something like this in 2 days was only possible thanks to great open-source library:

• Ogre3D (MIT)
• OpenCV (BSD)
• SiftGPU (non-profit license)

Analysis

I’m using a panoramic image as reference. For each frame of the video I’m extracting Sift feature using SiftGPU and matching them with those of the reference image. Then I’m computing the homography between the 2 images using Ransac homography estimator (OpenCV cvFindHomography).

Performance

The performance are low due to complexity of the Sift detection and matching and that I’m applying the homography using cvWarpPerspective.

I’m working on another version using Fast (or Agast) as feature detector and Brief as descriptor. This should lead to a significant speed-up and may eventually run on a mobile… Using the GPU vertex and pixel shader instead of the CPU to apply the homography should also gives a nice speed-up.

I’m also aware that it is not correct to apply an homography on a cylindric panoramic image (especially if you don’t undistort the input video frame too )

This demo is a ripp-off the Kinect-v11 demo made by Zephod. In fact I’ve designed a new Ogre::Kinect library that provide Kinect connection through Zephod library. Then I’ve replace the Zephod OpenGL demo by an Ogre3D demo using my library. The nice part is that I’ve managed to move some depth to rgb conversion to the GPU (using pixel shader).

Links

Binary demo: OgreKinectDemo1.zip
Source code: svn on code.google
Documentation: doxygen
License: MIT

I have finally released my Structure-From-Motion Toolkit (SFMToolkit). So what can you do with it ? Let’s say you have a nice place like the one just bellow:

Place de la Bourse, Bordeaux, FRANCE (picture from Bing)

Well, now you can take a lot of pictures of the place (around 50 in my case):

And then compute structure from motion and get a sparse point cloud using Bundler:

Finally you have a dense point cloud divided in cluster by CMVS and computed by PMVS2:

You can also take a loot at the PhotoSynth reconstruction of the place with 53 pictures and 26 (without the fountain).

This is the SFMToolkit workflow:

SFMToolkit is composed of several programs:

• BundlerFocalExtractor : extract CCD width from Exif using XML database.
• BundlerMatcher : extract and match feature using SiftGPU.
• Bundler : http://phototour.cs.washington.edu/bundler/ created by Noah Snavely.
• CMVS : http://grail.cs.washington.edu/software/cmvs/ created by Yasutaka Furukawa.
• PMVS2 : http://grail.cs.washington.edu/software/pmvs/ created by Yasutaka Furukawa.
• BundlerViewer : Bundler and PMVS2 output viewer based on Ogre3D (OpenSource 3D rendering engine).

Download

As you can see this “toolkit” is composed of several open-source component. This is why I have decided to open-source my part of the job too. You can download the source code from the SFMToolkit github. You can also download a pre-compiled x64 version of the toolkit with windows scripting (WSH) for easier usage (but not cross-platform): SFMToolkit1.zip.

Help

If you need some help or just want to discuss about photogrammetry, please join the photogrammetry forum created by Olafur Haraldsson. You may also be interested by Josh Harle’s video tutorials, they are partially out-dated due to the new SFMToolkit but these videos are very good to learn how to use MeshLab.

Please go to the SFMToolkit page to get the latest version

As you can see in this video I have managed to use PMVS2 with PhotoSynth output.
All the synths used in this video are available on my PhotoSynth account or directly:

• Porte Cailhau: http://photosynth.net/view.aspx?cid=1e509490-5657-453d-a2f6-2e55d14ae512
• Goutz: http://photosynth.net/view.aspx?cid=1471c7c7-da12-4859-9289-a2e6d2129319
• Place de la Bourse: http://photosynth.net/view.aspx?cid=e82eca65-60fe-498b-8916-80d1e3245640

Workflow

My PhotoSynth ToolKit is composed of 3 programs:

• PhotoSynthDownloader: download 0.json + bin files + thumbs
• PhotoSynth2PMVS: undistort bunch of pictures and write CONTOUR files needed for PMVS2
• PhotoSynthTileDownloader [optional]: download all pictures of a synth in HD (not relased yet for legal reason, but you can watch a preview video)

Limitations

It seems that my first version doesn’t handle the JSON parsing of all kind of synth very well, I’ll try to post a new version asap. fixed in PhotoSynthToolKit2.zip

PMVS2 for windows is a 32bit applications, so it has a 2Gb memory limits (3Gb if you start windows with the /3Gb options + compile the app with custom flag ?). I haven’t tried yet the 64bit linux version but I have managed to compile a 64bit version of PMVS2. My 64bit version manage to use more than 4Gb of memory for picture loading, but it crashes right after the end of all picture loading. I didn’t investigate that much, it should be my fault too, compiling the dependencies (gsl, pthread, jpeg) wasn’t an easy task.

Anyway, PMVS2 should be used with CMVS but I’m not sure that I can extract enough information from PhotoSynth. Indeed Bundler output is more verbose, you have 2d/3d correspondence + number of matches per images. I think that I can create a vis.dat file using some information stored in the JSON file but it should only speed-up the process, so it doesn’t help that much with the 2Gb limits.

Credits

My PhotoSynth ToolKit is coded in C++ and the source code is available on my google code (MIT license). It is using:

• Boost.Asio: network request for Soap + file download
• TinyXml: parsing of soap request
• JSON Spirit: parsing of PhotoSynth file: “0.json”
• jpeg: read/write jpeg for radial undistort

Furthermore, part of the code are based on:

• Bundler: RadialUndistort + Bundler2PMVS
• SynthExport: C# binary loader

Please go to the PhotoSynthToolkit page to get the latest version

So the idea is simple: the client uploads a video frame and the server does the pose estimation and send back the augmented rendering to the client. My first prototype is using ArToolKitPlus in almost real-time (15fps) but I’m also working on a markerless version that would be less interactive (< 1fps). The mobile client was an UMPC (Samsung Q1).

Thanks to Boost.Asio I’ve been able to produce a strong client/server very quickly. Then I have created two implementations of PoseEstimator :

class PoseEstimator
{
	public:
		bool computePose(const Ogre::PixelBox& videoFrame);
		Ogre::Vector3 getPosition() const;
		Ogre::Quaternion getOrientation() const;
}

• ArToolKitPoseEstimator (using ArToolKitPlus to get pose estimation)
• SfMPoseEstimator (using EPnP and a point cloud generated with Bundler -Structure from Motion tool- to get pose estimation)

ArToolKitPoseEstimator

There is nothing fancy about this pose estimator, I’ve just implemented this one as proof of concept and to check my server performance. In fact, ArToolKit pose estimation is not expensive and can run in real-time on a mobile.

SfMPoseEstimator

I’ll just introduce the concept of this pose estimator in this post. So the idea is simple, in augmented reality fake rolex you generally know the object you are looking at because you want to augment it. The idea was to create a point cloud of the object you want to augment (using Structure from Motion) and keep the link between the 3D points and theirs 2D descriptors. Thus when you take a shot of the scene you can compare the 2D descriptors of your shot with those of the point cloud and so create 2D/3D correspondence. Then the pose estimation can be estimated by solving the Perspective-n-Point camera calibration problem (using EPnP for example).

Performance

The server is very basic, it doesn’t handle client queuing yet (1 client = 1 thread), but it already does the off-screen rendering and send back the texture in raw RGB.

The version using ArToolKit is only Replica Handbag running at 15fps because I had trouble with the jpeg compression so I turn it off. So this version is only bandwidth limited. I didn’t investigate this issue that much because I know that the SfMPoseEstimator is going to be limited by the matching step. Furthermore I’m not sure that it’s a good idea to send highly compressed image to the server (compression artifact can add extra features).

My SfMPoseEstimator is also working but it’s very expensive (~1s using the GPU) and it’s not always accurate due to some flaws of my original implementation. I’ll explain how it works in my following post.

I have used the “bundle.out” file produced by Bundler to get cameras informations:

• intrinsic parameters: focal, distorsion
• extrinsic parameters: position, orientation

With these informations you can see the point cloud through the viewpoint of one of the camera registered by Bundler. I’ve added this feature to my current Ogre3D PlyReader. I also have added a background plane to be able to see the picture taken from this viewpoint. This demo is not available for download right now, but you can still watch the video :

The Ogre3D PlyReader and BundlerMatcher will eventually be added to my SVN. I’m currently busy working on another demo, so stay tuned !

There are three demos in this video:

1. GPU-Surf live demo.
2. PlyReader displaying Notre-Dame dataset.
3. PlyReader displaying my own dataset (Place de la Bourse, Bordeaux).

GPU-Surf

You’ll get more information on the dedicated demo section.
In this video GPU-Surf was running slowly because of Ogre::Canvas but it should be running really faster.

PlyReader displaying Notre-Dame dataset

I’m also interested in structure from motion algorithm, that’s why I have tested Bundler, which comes with a good dataset of Notre-Dame de Paris.

I have created a very simple PlyReader using Ogre3D, the first version was using billboard to display point cloud but it was slow (30fps with 130k points). Now I’m using custom vertex buffer and it runs at 800fps with 130k points.

The reconstruction was done by the team who created Bundler from 715 pictures of Notre-Dame de Paris (thanks to Flickr). In fact, in this demo they have done the big part of the job, I have just grab their output to check if my PlyReader was capable of reading such a big file.

PlyReader displaying my own dataset

If you already used Bundler you know that structure from motion algorithm needs a very slow pre-processing step to get “matches” between pictures of the dataset. Bundler is packaged to use Lowe’s Sift binary, but it’s very slow because it’s taking pgm as picture input and the output is written in a text file. Then a matching step is executed using KeyMatchFull.exe which is optimized using libANN but still very slow.

I have replaced the feature extraction and matching steps by my own tool: BundlerMatcher. It is using SiftGPU, which gives a very nice speed-up. As my current implementation of GPU-Surf isn’t complete I can’t use it instead of SiftGPU but this is my intention.

23 pictures taken with a classic camera (Canon Powershot A700)	Point cloud generated using Bundler

I have created this dataset with my camera and matched the pictures using my own tool: BundlerMatcher. This tool creates the same .key file as Lowe Sift tool and creates a matches.txt file that is used by Bundler. I have tried to get rid off this temporary .key file and keep everything in memory but changing Bundler code to handle this structure was harder than I predicted… I’m now more interested by insight3d implementation (presentation, source) which seems to be easier to hack with.

What is GPUSurf ?

It is a GPU accelerated version of Surf algorithm based on a paper of Nico Cornelis¹.
This version is using GPGPU technique (pixel shader) and Cuda for computing. The Cuda part was done using my Ogre::Cuda library and the GPGPU part was done using a new library called Ogre::GPGPU. This new library is just a little helper that hide the fact that GPGPU computing is done using quad rendering.

Screenshot of my GPU-Surf implementation (3 octaves displayed)

GPU-Surf could be used to help panoramic image creation, create tracking algorithm, speed-up structure from motion… I’m currently using SiftGPU to speed-up image matching step of structure from motion tools (bundler), but SiftGPU v360 as a memory issue (it’s eating a lot of virtual memory, your program as to be 64bit to bypass common 32bit application limitation: 2Go of virtual memory under windows), and Sift matching is more expensive than Surf (descriptor of 128 float vs 64 for Surf). That’s why I have decided to create my own implementation of GPU-Surf.

Implementation details:

The current version of my implementation of GPU-Surf is uncomplete (descriptor missing, only detector is available). You’ll get all information about license (MIT), svn repository, demo (GPUSurfDemo1.zip) and documentation on the GPUSurf page.

[1]: N. Cornelis, L. Van Gool: Fast Scale Invariant Feature Detection and Matching on Programmable Graphics Hardware (ncorneli_cvpr2008.pdf).