As you may have figured out I’m not working for acute3D anymore. I really enjoyed working there and I want to thanks them for the time we’ve spent together. I’ve learned a lot thanks to them and it was really great to deal with 3d data at a city scale!

BTW you should really checkout their online demos: streaming of 3d cities over internet!

But I’ve received an offer from Microsoft that I coudn’t refuse . I’ll be working on super exciting stuff closely related to what I like.

Thus sadly I don’t think that I’ll have time left for this blog anymore. But I’ll try to post the source code of my modified version of Bundler.

--parallel_epipolar
--ba sba

BTW those options should be passed as command arguments to bundler.exe directly (and not been added to the options.txt file).

The parallel_epipolar option allows to estimate the fundamental matrices (ransac + lm) in parallel. This problem is embarrassingly parallel but due to some global variables used in the callback passed to lmfit it wasn’t that easy to implement: I’ve used a functor to hide the global variables used by the callback but sadly lmfit doesn’t have a void* userData parameter (only a pointer to the callback). Thus I’ve modified lmfit and add this missing parameter that allow me to pass a pointer to my functor. Furthermore I had to compile almost everything in C++ instead of C to use my functor inside lmfit. Thus I had to fix a lot of malloc calls that weren’t compiling in C++ due to missing cast. To keep a “backward behavior compatibility” this option is disabled by default.

The ba option allows to change the bundle adjustment “engine” used. Here is the list of available “engine”:

• sba (default)
• none (for debug only)
• pba_cpu_double
• pba_cpu_float
• pba_gpu_float

Pba stand for Parallel Bundle Adjustment: I’ve integrated mcba from Changchang Wu.

So if you have an Nvidia GPU card and installed the Cuda runtime you can add those options:

bundler.exe list_focal_absolute.txt --ba pba_gpu_float --parallel_epipolar
--options_file options.txt //on the same line

Timing on a 245 pictures dataset:

You can download this new version of bundler: bundler-multiBA-parallelEpipolar-x64.zip

Nghiaho optimizations are disabled by default as I’ve found that it generates a point cloud less dense. You can enable the optimization by adding:

--optimize

as last PMVS2.exe parameter in your command prompt or just activate the optimization by modifying the following files:

• 3 – Prepare for PMVS2.wsf line 49 -> var USE_OPTIMIZED_PMVS = true;
• 4 – Prepare for CMVS.wsf line 46 -> var USE_OPTIMIZED_PMVS = true;

You can download PhotoSynthToolkit11.zip.

This version introduce a new feature that will allow you to process a PhotoSynth without needing to wait for the end of the upload. In fact you can even cancel your synth as soon as the computation is complete. But be aware that all pictures uploaded in the meantime will stay on photosynth server (AFAIK).

How to proceed:
- wait for synth to be processed (but not fully uploaded).
- copy the 2 following files from %temp%\Photosynther to another folder (d:\my_synth for example)

• collection.dzcz
• collection.synth.bin

- then you can launch the step 1 of the PhotoSynthToolkit by providing your folder (d:\my_synth).
- you’ll need then to copy your original jpegs to the distort folder and you’re done!

You can download PhotoSynthToolkit10.zip.

• CMVS support finally integrated
• proper handling of folder with white space

How to use CMVS with PhotoSynthToolkit:

1. As usual run step 1 and move pictures from “hd” to “distort” folder.
2. You need to run first “Prepare for PMVS2″ then do NOT launch the generated bat file.
3. You can run “Prepare for CMVS” and run the generated bat file.

Warning: I’ve fixed all my programs to handle folder with white space (thanks to boost::filesystem). Sadly PMVS2/CMVS don’t handle them. The fix is trivial but instead of recompiling a new version I’ve found a bad workaround. I’m creating a temporary virtual drive z: pointing to your synth folder during the execution of PMVS2/CMVS using SUBST. The downside are that if you already have a z: drive it won’t work and if you kill PMVS2/CMVS you’ll have to manually remove the z: drive. To remove the z: drive type “subst z: /d” in your console.

If you need XSI/3DS max integration you’ll have to use the old version: PhotoSynthToolkit7.zip otherwise PhotoSynthToolkit9.zip is the latest version.

Illustrations from Wikipedia
Input points	Estimated fitting line

->Launch live demo<-

Animated GIF of the iterations

New feature coming:

• CMVS support (coming very soon)
• White space handling in the toolkit scripting (PhotoSynthGrabber is already ready)
• Expose HTTP Proxy parameters for the downloader

You can download the new version: PhotoSynthToolkit8.zip or keep the previous one in case you miss some advanced feature (vis.dat generation, SoftImage XSI / 3DS Max scripting for example): PhotoSynthToolkit7.zip.

There is still room for improvements: I’ve discovered lot of interesting stuff that I’ve to try in Lilli Thompson speech at the GDC. In particular I’d like to improve:

• memory usage: track V8 garbage collection
• thumbnails loading: could be speed-up by using a web workers too
• Three.js rendering: I still need to switch to r48 and use BufferGeometry
• bundle.out generation: still very slow for big synth

To accelerate bundle.out file generation I’ve found a solution working pretty well for very large ascii file (80mo+). I’m generating the ascii file by writing directly in binary in a Uint8Array (by chunk of 1mo). The resulting code is more complex but is as fast as regular string concatenation and in comparison as a memory footprint very limited. It’s not integrated in the extension yet as my test case is working well (ascii ply generation) but bundle.out is slow (I need to track V8 bailout/deopt to understand why it’s slow).

I’ve also improved a lot my binary parsing by using only native binary parsing (DataView). One thing about DataView: did they realize that if you forgot the endianness parameter it will use BigEndian? (littleEndian = x86, bigEndian = Motora 68k).

Thanks to both optimization the new version is 4x faster! (loading time: 24s > 6s)

solution 1

• cross-orign was forbidden in contentscript (fixed now since chrome 13)
• lack of TypedArray copy/transfert -> JS array replacement
• Loading time: 13600ms (24400ms with the previous version without optimized binary parsing)

solution 2

• Issue: the UI is frozen during parsing (lack of threading)
• Loading time: 6500ms

solution 3

• I’m using transferable objects to both send the xhr.response (ArrayBuffer) and receive the parsed Float32Array (vertex positions) and Uint8Array (vertex colors).
• Issue: I didn’t manage to spawn a worker from my extension directly (SECURITY_ERR: DOM Exception 18): I’ve been forced to inline the worker as a workaround.
• Loading time: 6000ms

solution 4

• Issue: because of this bug workers can not do cross-orign xhr.
• Loading time: N/A

I’ve also optimized the PLY file generation by generating binary file on the client side instead of ascii.

• Ascii: 74mo in 9000ms
• Binary: 17mo in 530ms (~17x faster!)

Sadly I didn’t manage to find a way to accelerate “bundle.out” ascii file generation yet (involving lot of string concatenation and numbers formatting).

I didn’t have updated the extension on the chrome web store yet as I’m working on using the new BufferGeometry of Three.js that should be more efficient. Building BufferGeometry directly from Float32Array (vertex positions array) and Uint8Array (vertex colors array) seems a really better solution IMO.

Mesh compression

I’ve also discover the webgl-loader mesh compression solution recently. I’ve compiled my own version of objcompress.cc and fixed the JSON export. Then I’ve created my own format allowing to pack in one file all the utf8 files (thus reducing the number of http request needed). I will post about it soon: stay tuned!

Firefox extension?

On a side note I’ve also been playing with the new mozilla add-on builder SDK… well IMO Firefox isn’t the best browser for extension anymore. I’ve been very disappointed by the restriction applied to code running in the contentscript (for example prototype.js is not working: I had to use a clone that I’ve built long time ago). And I’m also very concerned about their security strategy: they don’t have a cross-orign xhr permissions field in the manifest. Thus extensions can do cross-orign request to all domains by default! Nevertheless I’ve managed to port the binary parsing of the point cloud but Three.js is not working either .

I’m sure that some of you have an idea of what to do with a bundle.out file from a PhotoSynth .
BTW, this export wouldn’t have been possible without Peter Sibley help, thanks Peter! He has help me by giving me the missing information needed to do the bundle.out PhotoSynth export.