Barco Labs 3-D mashups

Company
Barco Labs division of Barco NV
Location
Milpitas, CA
Start
September 2018
End
December 2018

After completing Graphical Content Analysis server for the ISE Expo, the idea of a common operational picture came up. That is, the user is presented with one view for everything where one can seamlessly zoom out to see the big picture or zoom in on a specific area or incident to see those details without ever switching views. And, it just so happened that the Antwerp Port Authority had recently suffered millions in damages due to a large barge colliding into one of the canal locks because someone wasn't paying attention. So, the Port Authority was interested in possibly using our anomaly detection technology to monitor canal surveillance cams. Since the Antwerp canal system covers such a large area we thought that combining anomaly detection with 3-D situational awareness was worth some research.

I started off by doing a detailed survey of the state-of-art in 3-D rendering technologies with an emphasis on web delivery. I looked at BabylonJS, ThreeJS, OpenSceneGraph, Open Cobalt, Lumion 3D, SketchUp, ESRI, deck.gl and vis.gl from Uber, Mapbox 3D, OpenStreetMaps 3D, the Unity and Unreal game engines, and other technologies (that I no longer recall).

I was intrigued by the fact that the Unity and Unreal game engines had recently started supporting Mozilla's WebAssembly technology which allows interactive 3-D gaming within web browsers without the need for 3rd-party plug-ins. At the time, this was all very experimental. It looked like we might leverage this game-engine web-delivery as a 3-D situational awareness app using existing game model assets combined with the surveillance cam feeds together with our anomaly detection technology. This Port of Antwerp browser prototype using MapBox and OpenStreetMaps 3-D shows more or less what we had in mind within the Pixel Analytics web UI.

Port of Antwerp 3-D

Live surveillance cam videos (shown as billboards) and their locations could be mashed up together on a 3-D terrain map. Zooming would zoom the video billboards proportionately. So, one could fly into a location to see what was happening better. We also had the idea of "smart touring" where the system would automatically "fly" to each surveillance location in turn unless an anomaly was detected, in which case, the system would "fly" to the location where the anomaly occurred. Other overlays could also be mashed up into the scene.

However, both the Mabbox terrain map and OpenStreetMaps 3-D assets were 2-years out-of-date. A whole new canal complex had been built and this view only shows that the land had been cleared for construction. So, to bring things up-to-date, it looked like the missing infrastructure might be filled in using game model assets (such as roads, waterways, bridges and ships). So, I spent more time looking at the Unity and Unreal game engines. Both game engines use Mozilla's Emscripten technology to build WebAssemblies.

One thing that became obvious was that WebAssembly does not support WebRTC video. This was because Emscripten uses OpenGL ES and there is no support for WebRTC in OpenGL ES. Live WebRTC video from multiple surveillance cams was a must have which might have thrown the whole WebAssembly idea out the window except web browsers support WebRTC in WebGL and WebAssemblies run inside web browsers. I was hoping it might be possible to get WebRTC video from JavaScript to work within WebAssemblies with a bit of trickery.

So, that led to building a prototype to see if it was possible to get WebRTC video to display within WebAssemblies. Since the game engines use Emscripten, I did the same. The only web browsers that supported WebAssembly, as an experimental feature at the time, were Firefox and Chrome. Emscripten source code must be written in C or C++ mixed with OpenGL ES shader code for 3-D rendering (usually with GPU acceleration for a richer gaming experience). The C++ code is then compiled to assembler byte code by LLVM and saved in a WASM file that can be loaded by a compatible web browser. Chrome additionally has a just-in-time compiler built-in that can compile the LLVM byte code into native object code specific to the OS/CPU the browser is running on. Thus complied WebAssembly runs as native code and that's a lot faster than interpreted JavaScript.

After a bit or trial and error, I did successfully develop a WebAssembly prototype that could show WebRTC video from a live web cam rendered as a 2-D OpenGL texture on a rotating OpenGL object in 3-D. I don't think anyone else had ever done this before. The OpenGL pipeline with WebRTC video included from the JavaScript code looks like this.

WebRTC video included within WebAssembly

Emscripten provides a way of calling JavaScript from WebAssembly code and vs. versa. These inter-op call mechanisms were essential for pulling this off. WebGL has been extended to take a 2-D HTML canvas rendered from WebRTC (via the WebGL texImage2D call) and include that video frame texture within WebGL. OpenGL ES lacks that feature but all OpenGL ES calls are actually routed through WebGL due to WebAssembly's sandboxing. (A 3-D graphics context for WebAssembly is created by _glutCreateWindow on the Javscript side and passed back to the WebAssembly code during initialization.) So, the trick was to call JavaScript from WebAssembly (using the Emscripten ES_ASM macro) which executes the WebGL texImage2D call using the graphics context that the WebAssembly code was using for OpenGL ES. Here is the code snippet for that call.

void loadTextureDynamic() {
  // There is no special version of the WebGL glTexImage2D that can read HTML
  // elements directly. So, we need to use an Emscripten inter-op call to do
  // this in JavaScript/WebGL.
  EM_ASM(getVideoTexture());
  // emscripten_run_script("getVideoTexture()");
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
}

getVideoTexture() on the JavaScript side was thus.

function getVideoTexture() {
  const level = 0;
  const internalFormat = Module.ctx.RGBA;
  const srcFormat = Module.ctx.RGBA;
  const srcType = Module.ctx.UNSIGNED_BYTE;
  // WebGL has special versions of texImage2D linked to HTML 5 canvas elts.
  Module.ctx.texImage2D(Module.ctx.TEXTURE_2D, level, internalFormat,
                        srcFormat, srcType, videoElt);
}

Because the GC is now shared on both sides, JavaScript could then pass back the WebRTC video texture to the WebAssembly for inclusion within the OpenGL ES 3-D scene. With the success of this prototype I was given the green light to explore the Unreal and Unity game engines further.

I compared both game engines with a focus on WebAssembly. The Unreal game engine turned out to have the edge in this regard. Here are the main reasons.

Programming languages
Unity game code is written in C# and Unreal game code is written in C++. Emscripten also must be in C++. This alone gave Unreal the upper edge because Unity had to transcode C# to C++ for input into Emscripten. While there was a Unity API for linking code compiled from C++, it was all rather indirect. The other plus for Unreal was that I was just more comfortable coding C++ than C#.
Open space rendering
Unreal did open-space rendering better than Unity at that time. Unity was more often used for 2-D games with great animation support while Unreal was better known for producing beautiful 3-D scenes with great landscape rendering support. And the latter was what we were really after.
Graphical modeling
Unreal had a graphical model builder called Blueprints that Unity lacked at the time (except via a possible 3rd-party plug-in). This was important because we didn't want to be in the business of building custom solutions for customers. A graphical builder tool like Blueprints seemed like a good way to provide the building blocks that 3rd-party integrators could leverage. That way, Barco could sell a generic product to integrators who would in turn use our Blueprint modules combined with Unreal's to create custom solutions for their customers. And, the more I looked at Unreal's Blueprint builder the more I liked it.

That all added up to giving Unreal the edge. The Blueprints model builder had a number of similarities to the Node-RED data flow builder we previously used in the GCA-EAS project. Based on that, I mocked up a possible set of Blueprint modules that would make building custom solutions fairly straightforward. Here is the mockup.

Unreal Blueprint mockup

I then played around with the Unreal engine learning how to build scenes, import terrain maps, add water effects, etc. Then, I tried my hand at building Blueprints modules via their C++ Blueprints API. Here are some of my custom Blueprint modules wired together.

Blueprint mashup modules

That screen shot is a bit difficult to read so here is the left portion of the screen shot above.

Blueprint mashup modules left

And, here is the middle portion of the same Blueprints screen.

Blueprint mashup modules middle

While these are actual custom Blueprints modules written in C++, I never really had a chance to fill these modules in with real functioning code. We gave our stakeholders a demo of where we were going and they felt this project would compete with a product in this space from one of our partners. The stakeholders felt it was more important to maintain a good working relationship with our partner than compete against them. Therefore, this project was put on hold. But, it was a lot of fun while it lasted. And, I am continuing to play with Unreal and WebAssembly on my own time.