You can get the extension here. Make sure you are logged in on photosynth.net (and have joined the beta) while using the extension as some photosynth2 feature are still only available to beta users (hopefully this limitation will be removed soon).

Feature added by the extension:

The extension is changing the top menu to:

• Search: previous Explore page + new links to fullscreen grid based ps2 explore page.
• Explore: new fullscreen map based explore page.
• My Synths: previous My Photosynths page + new tab (my map, my ps2, my info).
• Preview: link to the new preview website + animated preview on the create page.

Search:

Animated preview when you hover a thumbnail of a PS2 + links to new explore page.

Clicking on “Explore most recent PS2″ or “Explore most viewed PS2″ will display a page like this:

The bottom row contains the latest synths of the current user (the one of the current synth being played). You can hover to see a preview and click to load a synth. Experimental feature: it’s using a new version of the viewer capable of quickly unload/load synths (don’t click too fast ).

Explore:

New fullscreen page showing synths (ps1, panorama and ps2) on a map. Sadly there is no proper ranking here so you need to zoom in a lot before being able to see synths. Also there is a known bug in google chrome which is preventing from seeing silverlight content opened in new tab.

My Synths:

The extension add 3 new tabs: My map, My PS2, My info

My map

This page is really complicated, adding a lot of feature… I will propably have another post explaining how to use it.

Basically you can map (=geotag) your own synths (ps1, panorama, ps2). You need to be logged in and click on ‘show untagged synths’ then you can search for a place on the map, move the orange pushpin or right click to reassign it location. Then once the orange pushpin is at the place where you’ve captured your synth, you can click on ‘assign pushpin location’ to assign the location to the corresponding synth. To select synths, you can press and hold ‘ctrl’ and then draw a rectangle, from that selection you can either remove their map position or assign them a text tag. Clicking on ‘movable pushpins’ will allow you to directly move your synth pushpins on the map. You can click on on ‘Find neighbors’ to find synths arround your synth (orange synth = community, green = yours). You can also play with the timeline to only display synths captured in the corresponding time interval. You can also change the url and switch w=0 to w=1 to enable the wikipedia option. This option will search for the closest wikipedia of your synth. Please consider donating to WikiLocation if you are using the w=1 option.

My PS2

This is a new page allowing you to quickly preview all your ps2 synths. They are grouped by topology (spin, panorama, wall, walk) and then sorted by captured date.

My info

Sadly the captured date information is not properly filled by the system (it’s using the upload date). You need to click on ‘fix capturedDate’ to set the captured date of all your ps2 (it might take a while: wait for the ‘done’ alert box).

Preview – Create

The extension is adding an animated preview which will help you to choose the proper topology:

Comments

This extension was created by me only and thus it doesn’t mean that this is representative of upcoming photosynth feature.

FYI this is not my first extension for the photosynth website, I’ve already created one which is adding a webgl fallback viewer if you don’t have silverlight for photosynth1 content.

Have fun exploring all feature introduced in this chrome extension!

This is a short post in case you’ve missed the launch of Photosynth2. I’m glad to be able to break the silence and to show some nice stuff again . I’ve developed the WebGL viewer used to display ps2 synths and I’ve been busy capturing tons of ps2′s (I still have a lot of datasets waiting to be uploaded ).

If you are using a modern browser you should be able to see this ps2 with the WebGL viewer:

Otherwise we are also generating an mp4 fallback:

In case you are wondering what’s new compared to photosynth1, I’ve created this video showing the benefit of a ‘good’ geometry (ps2) for image-based rendering instead of using only a dominant plane (ps1):

Photosynth 1 is allowing unstructured capture and thus the navigation is very difficult. In Photosynth 2 we are constraining the user to capture images along a single path (1D manifold) and thus the navigation is very simple (and touch friendly ). Photosynth 2 introduces 4 different topology (camera motions):

Spin:

Panorama:

Walk:

Wall:

BTW if you type: “The answer to life, the universe and everything”, on the view page of a synth a magic menu will appear with lot of options so that you can tweak your viewer. You will need to refresh the view page to apply most options and the changes are permanent (store in localStorage).

‘m’ and ‘c’ are also two other shorcuts that you should try…

I should have another post very soon. Stay tuned!

Live Demo (require WebGL enabled browser)

My implementation is very strait-forward: my PhotoSynthHDDownloader creates 6 jpg from a panorama url. The 6 faces of the cube are downloaded at full resolution. This solution is suitable for panorama created with an iPhone (low res) but not acceptable for gigapixel panorama (you need progressive download/display). I’m using Three.js as WebGL engine and I’d like to implement a megatexture solution to display very large panorama (I’ve found a very impressive working implementation and played with it: it is working with Google Chrome / Firefox 10).

To be able to handle panorama as well, my Google Chrome extension needs to use megatexture and it would be really more efficient if Microsoft could add the missing cors header to allow WebGL cross-origin (even only for photosynth.net). And as they’ve just moved their data to Windows Azure, I guess that it may be easier to add this header . Otherwise I’ll have to use my special trick to bypass webgl same-origin-policy restriction.

2010 was a year full of visual experiments for me, I hope that you like what you see on this blog. In this post I’m making a little overview of all visual experiments created by me during this year. This is an opportunity to catch-up something you’ve missed! I’d like also to thanks some person that have been helping me too:

• Olafur Haraldsson: for creating the photogrammetry forum
• Josh Harle: for his videos tutorials and his nice blog
• You: for reading this

Visual experiments created in 2010:

During this year I have added some features to Ogre3D:

• ArToolKitPlus: augmented reality marker-based system
• Cuda: for beginner only (at least advanced user could grab some useful code)
• OpenCL: for beginner only (at least advanced user could grab some useful code)
• Html5 Canvas: implementation based on skia for graphics and V8 for javascript scripting
• Kinect: this is a very hacky solution, I’ll improve it later

I also have learned GPGPU programming by myself while coding a partial GPUSurf implementation based on Nico Cornelis paper. But this implementation is not complete and I’m willing to rewrite it with a GPGPU framework based on OpenGL and CG only (not Ogre3D). With such a framework writing Sift/Surf detector should be easier and more efficient.

I have created some visual experiments related to Augmented Reality:

• Remote AR prototype
• Outdoor 3D tracking using point cloud generated by structure from motion software
• Outdoor 2D tracking using panorama image

My outdoor 3D tracking algorithm for augmented reality needs an accurate point cloud: this is why I’m interested in structure from motion and I’ve created two SfM toolkit:

• SFMToolkit (SiftGPU -> Bundler -> CMVS -> PMVS2)
• PhotoSynthToolkit (PhotoSynth -> PMVS2)

Posts published in 2010:

• 2010/12/22: Outdoor tracking using panoramic image
• 2010/12/20: Structure from motion projects
• 2010/12/13: Augmented Reality outdoor tracking becoming reality
• 2010/11/20: Kinect experiment with Ogre3D
• 2010/11/19: PhotoSynthToolkit results
• 2010/11/09: PhotoSynth Toolkit updated
• 2010/11/05: Structure From Motion Toolkit released
• 2010/09/27: My 5 years old Quiksee competitor
• 2010/09/23: PMVS2 x64 and videos tutorials
• 2010/09/08: Introducing OpenSynther
• 2010/08/22: Dense point cloud created with PhotoSynth and PMVS2
• 2010/08/19: My PhotoSynth ToolKit
• 2010/07/12: Pose Estimation using SfM point cloud
• 2010/07/11: Remote Augmented Reality Prototype
• 2010/07/08: Structure From Motion Experiment
• 2010/06/25: GPU-Surf video demo
• 2010/06/23: GPUSurf and Ogre::GPGPU
• 2010/06/20: Ogre::Canvas, a 2D API for Ogre3D
• 2010/05/09: Ogre::OpenCL and Ogre::Canvas
• 2010/04/26: Cuda integration with Ogre3D
• 2010/04/09: Multitouch prototype done using Awesomium and Ogre3D
• 2010/03/05: ArToolKitPlus integration with Ogre3D
• 2010/02/20: Hello World !

Analysis

In a pre-processing step I’ve built a sparse point cloud of the place using my SFMToolkit. Each vertex of the point cloud has several 2D Sift features correspondences. I’ve only kept one Sift descriptor per vertex (mean of the descriptors) and put all descriptors in an index using Flann.

For each frame of the video to be augmented, I’ve extracted Sift feature with SiftGPU and then matched them using Flann 2-nearest neighbor search and a distance ratio threshold. The Flann matching is done in parallel with boost::threadpool. The matches computed contains a lot of outliers. So I have implemented a Ransac pose estimator using EPnP that permits to filter bad 2d/3d correspondences.

Performance

My implementation is slow (due to my implementation of Ransac EPnP that could be improved).

The slowness is not a so big issue because it doesn’t need to run at 30fps. Indeed the goal of my prototype is to have absolute pose with this tracking system each second and relative pose using inertial system available on mobile device (or using KLT tracking).

Issue

• Performance (faster is better )
• Point cloud reference is not always accurate (Bundler fault)

In another post I’ll introduce alternative to Bundler: faster and more accurate.

I’ve finally managed to spend a couple of hours to compile a 64 bit version of PMVS2 for windows! You can download PMVS2_x64.zip right now. I’ll hope that this version will help some persons, I’ve personally managed to create a very dense model thanks to this version and PMVS2 was using more than 4Gb of ram on a 8-cores machines.

How to compile PMVS2 x64 by yourself:
download the CMake package of CMVS (containing PMVS) created by Pierre Moulon.
download and compile gsl 1.8
download precompiled pthread x64 lib from equalizer svn
download and compile clapack 3.2.1 using CMake

PhotoSynthTileDownloader

As requested by some persons, I’ve updated my PhotoSynthTileDownloader: you can now resume a partial download ! It’s already available for download: PhotoSynthTileDownloader2.zip.

Videos tutorials

Josh Harle has done some very nice videos tutorials on how to use my PhotoSynth ToolKit and has created another toolkit for Bundler that is using my BundlerMatcher.

PhotoSynth Toolkit post on Josh Harle’s blog.

Note: In fact your synths doesn’t need to be 100% synthy. My tool (PhotoSynth2PMVS) is capable of using an uncomplete synth. And now you could use my 64bit version of PMVS2 instead.

Please go to the PhotoSynthToolkit page to get the latest version


Bundler photogrammetry package post on Josh Harle’s blog.

Please go to the SFMToolkit page to get the latest version

In my previous post I have released my PhotoSynth Tookit but the PhotoSynth tile downloader wasn’t available yet. You can now download the picture of your Synth in HD using PhotoSynthTileDownloader1.zip. But please respect author’s copyright ! I have included a new confirmation dialog box that warn you about the Synth status (unlisted, public) but this is not shown in the video presentation of PhotoSynthTileDownloader.

BundlerMatcher

I have finally released BundlerMatcher, a feature extraction and picture matching tool built with SiftGPU. The main goal of this tool was to replace the slow matching step packaged with Bundler by a faster one using GPU without needing to modify Bundler’s code. You can download BundlerMatcher1.zip or checkout the code on my google code. Warning: this tool needs a 64bit windows OS (tested on 7 and vista) and a decent GPU. All my demos are available under MIT license but SiftGPU isn’t released under MIT so you should take a look at SiftGPU license.

OpenSynther

I’m proud to introduce OpenSynther which is under heavy development. The first goal of this tool is the same as BundlerMatcher: provide a faster matching engine for Structure from Motion tools. To achieve high performance it is coded in C++ and using both multi-core and GPU. Furthermore OpenSynther is using Surf instead of Sift and may in the future also compute the 3d reconstruction (as PhotoSynth does).

OpenSynther current feature list:

• Jpeg loading + Exif reading
• multi-threaded Surf feature extraction (based on OpenSurf & ParallelSurf)
• multi-threaded Surf feature matching (based on OpenSurf matching + Cuda GPU matching)
  • Quadratic complexity O(n²) with n number of pictures

OpenSynther TODO:

• multi-threaded Surf feature approximate matching (based on FLANN)
  • Linear complexity (hypothetical)
• integrate 3d reconstruction ?
  • using Bundler code or insight3D, both are using SBA.

Benchmarks

These benchmarks were done on an Intel Corei7 920 (8 cores @ 2.66Ghz) + Nvidia GTX 285. The pictures size were 1600×1200 (~2M pixels), I heard that it was the limit size at which PhotoSynth extract features (it can of course use bigger pictures but feature extraction are done at a max size of 2M pixels).

Note:

• Bundler: I had to reduce picture to 640×480, otherwise I got too many features (would be unfair).
• PhotoSynth: I didn’t take into account the time spent in tile creation.
• OpenSynther: in the future I may consider using GPUSurf for feature extraction.
• BundlerMatcher: to be fair with PhotoSynth I didn’t write the ascii key file (so in fact it’s slower).

Note:

• PhotoSynth: they did a very good job and managed to have a linear matching !
• OpenSynther: the GPU is in fact doing the most part of the job as you can see in the table below:

As you can see OpenSynther matching is in fact done by the GPU at 85%.