I’ve spent 2 days implementing a WebService in C++. I’ve lost 1 day trying to use WebSocket (this wasn’t a good idea as Firefox and Google Chrome were not implementing the same version). I’ve also lost a lot of time with the beautiful drag and drop API… who is responsible for this mess? Anyway I’ve managed to create this cool demo:
The principle is very simple: the user drag and drop an unknown picture on the webpage. The picture is resized on the client side using canvas and then sent to the webservice using a classic POST Ajax call with the picture sent in jpeg base64. Then the pose estimator web service try to compute the pose and send an xml response. Then the pose is applied to the Three.js camera used in my PhotoSynth viewer.
I have cheated a little as I’ve used shots of a video with known intrinsic parameters. But I’ve two options to handle really unknown pictures:
If the jpeg as Exif I can use it to find the focal (using a CCD database and a JS Exif parser)
I hope you enjoy this demo as much as I do and that Google and Microsoft are working in that direction too (especially Microsoft with Read/Write World…)
I’m finally back from holiday and found enough time to update the Google Chrome extension with my new PhotoSynth Viewer based on Three.js.
You can download the new Google Chrome extension from the chrome web store: Three.js Google Chrome extension. The old one is still available but this is not a good idea to have both activated at the same time… Do not hesitate to use the donate button (located in the right side-panel) if you find this extension useful… thanks!
Warning: this extension may use a lot of ram (point cloud are store in memory as regular js array not TypedArray).
Possible improvement
Add picture thumbnails in the 3d environment (may become impossible without Microsoft adding CORS to the jpeg due to WebGL security issue)
Add a free flight camera (easy to add but harder without roll: I’ve added features to Three.js Quaternion class for this purpose, but still not ready)
Make the download point cloud link work! This is not my fault (I hope) I think that this is a Chrome bug. The only workaround that I can think of (without being sure that it would work) for this issue would be that Microsoft host downloadify.swf on photosynth.net… please ask them to do that!
Add a Meshlab like camera system (mouse handling + white sphere): if you know Three.js please give it a try!
I wasn’t happy with my previous WebGL PhotoSynth Viewer: my javascript parser was aware of the position of the cameras and I wasn’t doing anything useful with it… Furthermore I was only relying on XB-PointStream for the rendering (which manage to display perfectly a dense point cloud). But I definitively want to do more fancy stuff. So Goodbye XB-PointStream, Hello Three.js!
The only trouble with this demo is that it is using a proxy PHP page to bypass ajax cross-domain security issue. Thus I won’t put it online (as it would kill my bandwidth limit). But I’ll try to update my Google Chrome extension so that you can give it a try too (will take time and I’m going in holiday…sorry).
Accordion(except css) + Tween + SynthParser created by me
Other news
On another side I’ve been up to:
compile a bundler version with PBA support (with bad result: fast BA but erroneous point cloud)
create a new framework for fast matching: already 3x faster than OpenSynther (which is already really fast). This new framework is compatible with multi-gpu and multi-cpu + you can add constraints to memory usage!
create an hybrid AR solution for my current work based on my panorama tracking demo and using gyroscope information to fix the computing time of the vision part
I’ve worked a lot on OpenSynther lately: OpenSynther is the name of my structure-from-motion solution. This new version is a major rewrite of the previous version which was using Surf with both GPU and multi-core CPU matching. The new version is using SiftGPU and Flann to achieve linear matching complexity of unstructured input as described in Samantha paper. You can find more information about OpenSynther features on it dedicated page (including source code).
OpenSynther has been designed as a library (OpenSyntherLib) which has already proven to be useful for several programs written by myself:
OpenSynther: work in progress… used by my augmented reality demo
BundlerMatcher: this is the matching solution used by SFMToolkit
Outdoor augmented reality demo using OpenSynther
I’ve improved my first attempt of outdoor augmented reality: I’m now relying on PhotoSynth capability of creating a point cloud of the scene instead of Bundler. Then I’m doing some processing with OpenSynther and here is what what you get:
You can also take a look at the 3 others youtube videos showing this tracking in action around this church: MVI_6380.avi, MVI_6381.avi, MVI_6382.avi.
PhotoSynth2CMVS
This is not ready yet, I still have some stuff to fix before releasing it. But I’m already producing a valid “bundle.out” file compatible with CMVS processing from PhotoSynth. I’ve processed the V3D dataset with PhotoSynth2CMVS and sent the bundle.out file to Olafur Haraldsson who has managed to create the corresponding 36 million vertices point cloud using CMVS and PMVS2:
The new unstructured linear matching is really fast as you can see on the above chart compared to PhotoSynth. But the quality of the generated point cloud is not as good as PhotoSynth.
This benchmark was computed on a Core i7 with an Nvidia 470 GTX. I’ve also compared the quality of the matching methods implemented in OpenSynther (linear VS quadratic). I’ve used Bundler as a comparator with a dataset of 245 pictures:
Linear
Quadratic
Nb pictures registered
193
243
Time spent to register 193 pictures
33min
1h43min
On the one hand, both the matching and the bundle adjustment are faster with linear matching but on the other hand, having only 193 out of 245 pictures registered is not acceptable. I have some idea on how to improve the linear matching pictures registering ratio but this is not implemented yet (this is why PhotoSynth2CMVS is not released for now).
Future
I’ve been playing with LDAHash last week and I’d like to support this in OpenSynther to improve matching speed and accuracy. It would also help to reduce the memory used by OpenSynther (by a factor 16: 128 floats -> 256bits per feature). I’m also wondering if the Cuda knn implementation could speed-up the matching (if applicable)? I ‘d also like to restore the previous Surf version of OpenSynther which was really fun to implement. Adding a sequential bundle adjustment (as in bundler) would be really interesting too…
Off-topic
I’ve made some modifications to my blog: switched to WordPress 3.x, activated page caching, added social sharing buttons and added my LinkedIn account next to the donate button…
I'm Henri Astre, a French guy who likes visual experiments. I've done some nice demos in various domains (augmented reality, GPU computing, structure from motion, 3d rendering, web app...) that you can see on this blog. [+ resume]
(located in the right side-panel) if you find this extension useful… thanks!
