WebRTCWindowSystem is a BitmapWindowSystem with a WebRTC server that sends video frames to remote clients for visualization. More...

#include <WebRTCWindowSystem.h>

Inheritance diagram for open3d::visualization::webrtc_server::WebRTCWindowSystem:

Data Structures
struct	Impl

Public Member Functions
virtual	~WebRTCWindowSystem ()
OSWindow	CreateOSWindow (gui::Window o3d_window, int width, int height, const char title, int flags) override
void	DestroyWindow (OSWindow w) override
std::vector< std::string >	GetWindowUIDs () const
	List available windows.
std::string	GetWindowUID (OSWindow w) const
OSWindow	GetOSWindowByUID (const std::string &uid) const
void	StartWebRTCServer ()
	Start WebRTC server in a background thread.
std::string	OnDataChannelMessage (const std::string &message)
void	RegisterDataChannelMessageCallback (const std::string &class_name, const std::function< std::string(const std::string &)> callback)
void	OnFrame (const std::string &window_uid, const std::shared_ptr< core::Tensor > &im)
	Server -> client frame.
void	SendInitFrames (const std::string &window_uid)
std::string	CallHttpAPI (const std::string &entry_point, const std::string &query_string="", const std::string &data="") const
	Call PeerConnectionManager's web request API.
void	EnableWebRTC ()
void	DisableHttpHandshake ()
void	CloseWindowConnections (const std::string &window_uid)
	Close all WebRTC connections that correspond to a Window.
Public Member Functions inherited from open3d::visualization::gui::BitmapWindowSystem
	BitmapWindowSystem (Rendering mode=Rendering::NORMAL)
	~BitmapWindowSystem ()
void	Initialize () override
void	Uninitialize () override
void	SetOnWindowDraw (OnDrawCallback callback)
void	WaitEventsTimeout (double timeout_secs) override
Size	GetScreenSize (OSWindow w) override
void	PostRedrawEvent (OSWindow w) override
void	PostMouseEvent (OSWindow w, const MouseEvent &e)
void	PostKeyEvent (OSWindow w, const KeyEvent &e)
void	PostTextInputEvent (OSWindow w, const TextInputEvent &e)
bool	GetWindowIsVisible (OSWindow w) const override
void	ShowWindow (OSWindow w, bool show) override
void	RaiseWindowToTop (OSWindow w) override
bool	IsActiveWindow (OSWindow w) const override
Point	GetWindowPos (OSWindow w) const override
void	SetWindowPos (OSWindow w, int x, int y) override
Size	GetWindowSize (OSWindow w) const override
void	SetWindowSize (OSWindow w, int width, int height) override
Size	GetWindowSizePixels (OSWindow w) const override
void	SetWindowSizePixels (OSWindow w, const Size &size) override
float	GetWindowScaleFactor (OSWindow w) const override
float	GetUIScaleFactor (OSWindow w) const override
void	SetWindowTitle (OSWindow w, const char *title) override
Point	GetMousePosInWindow (OSWindow w) const override
int	GetMouseButtons (OSWindow w) const override
void	CancelUserClose (OSWindow w) override
void *	GetNativeDrawable (OSWindow w) override
rendering::FilamentRenderer *	CreateRenderer (OSWindow w) override
void	ResizeRenderer (OSWindow w, rendering::FilamentRenderer *renderer) override
MenuBase *	CreateOSMenu () override
Public Member Functions inherited from open3d::visualization::gui::WindowSystem
virtual	~WindowSystem ()

Static Public Member Functions
static std::shared_ptr< WebRTCWindowSystem >	GetInstance ()

Additional Inherited Members
Public Types inherited from open3d::visualization::gui::BitmapWindowSystem
enum class	Rendering { NORMAL , HEADLESS }
using	OnDrawCallback
Public Types inherited from open3d::visualization::gui::WindowSystem
using	OSWindow = void*
Static Public Attributes inherited from open3d::visualization::gui::WindowSystem
static constexpr int	FLAG_VISIBLE = 0
static constexpr int	FLAG_HIDDEN = (1 << 0)
static constexpr int	FLAG_TOPMOST = (1 << 1)

Detailed Description

WebRTCWindowSystem is a BitmapWindowSystem with a WebRTC server that sends video frames to remote clients for visualization.

WebRTCWindowSystem owns a PeerConnectionManager, which manages all things related to the WebRTC connections, e.g. get media lists, get and add ICE candidates, connect to a media and hangup.

When the client visit a Open3D visualizer's website for visualization (a.k.a. standalone mode), an HTTP handshake server will be used to serve the website and perform handshake to establish the WebRTC connection. In Jupyter mode, the HTTP handshake server is disabled and the handshake is done via Jupyter's JavaScript<->Python communication channel.

WebRTCWindowSystem shall be used as a global singleton. Both the PeerConnectionManager and the HTTP handshake server runs on different threads.

Constructor & Destructor Documentation

◆ ~WebRTCWindowSystem()

open3d::visualization::webrtc_server::WebRTCWindowSystem::~WebRTCWindowSystem ( )

virtual

Member Function Documentation

◆ CallHttpAPI()

std::string open3d::visualization::webrtc_server::WebRTCWindowSystem::CallHttpAPI	(	const std::string &	entry_point,
		const std::string &	query_string = "",
		const std::string &	data = "" ) const

Call PeerConnectionManager's web request API.

This function is called in JavaScript via Python binding to mimic the behavior of sending HTTP request via fetch() in JavaScript.

With fetch: data = {method: "POST", body: JSON.stringify(candidate)}; fetch("/api/addIceCandidate?peerid=" + peerid, data);

Now with CallHttpAPI: open3d.visualization.webrtc_server("/api/addIceCandidate", "?peerid=" + peerid, data["body"]);

Parameters

entry_point	URL part before '?'.
query_string	URL part after '?', including '?'. If '?' is not the first character or if the string is empty, the query_string is ignored.
data	JSON-encoded string.

◆ CloseWindowConnections()

void open3d::visualization::webrtc_server::WebRTCWindowSystem::CloseWindowConnections ( const std::string & window_uid )

Close all WebRTC connections that correspond to a Window.

◆ CreateOSWindow()

WebRTCWindowSystem::OSWindow open3d::visualization::webrtc_server::WebRTCWindowSystem::CreateOSWindow	(	gui::Window *	o3d_window,
		int	width,
		int	height,
		const char *	title,
		int	flags )

overridevirtual

Reimplemented from open3d::visualization::gui::BitmapWindowSystem.

◆ DestroyWindow()

void open3d::visualization::webrtc_server::WebRTCWindowSystem::DestroyWindow ( OSWindow w )

overridevirtual

Reimplemented from open3d::visualization::gui::BitmapWindowSystem.

◆ DisableHttpHandshake()

void open3d::visualization::webrtc_server::WebRTCWindowSystem::DisableHttpHandshake ( )

HTTP handshake server is enabled by default. In Jupyter environment, call DisableHttpHandshake() before StartWebRTCServer().

◆ EnableWebRTC()

void open3d::visualization::webrtc_server::WebRTCWindowSystem::EnableWebRTC ( )

Sets WebRTCWindowSystem as the default window system in Application. This enables a global WebRTC server and each gui::Window will be rendered to a WebRTC video stream.

◆ GetInstance()

std::shared_ptr< WebRTCWindowSystem > open3d::visualization::webrtc_server::WebRTCWindowSystem::GetInstance ( )

static

◆ GetOSWindowByUID()

WebRTCWindowSystem::OSWindow open3d::visualization::webrtc_server::WebRTCWindowSystem::GetOSWindowByUID ( const std::string & uid ) const

◆ GetWindowUID()

std::string open3d::visualization::webrtc_server::WebRTCWindowSystem::GetWindowUID ( WebRTCWindowSystem::OSWindow w ) const

◆ GetWindowUIDs()

std::vector< std::string > open3d::visualization::webrtc_server::WebRTCWindowSystem::GetWindowUIDs ( ) const

List available windows.

◆ OnDataChannelMessage()

std::string open3d::visualization::webrtc_server::WebRTCWindowSystem::OnDataChannelMessage ( const std::string & message )

Client -> server message.

Returns: Message reply that should be sent back to the client.

◆ OnFrame()

void open3d::visualization::webrtc_server::WebRTCWindowSystem::OnFrame	(	const std::string &	window_uid,
		const std::shared_ptr< core::Tensor > &	im )

Server -> client frame.

◆ RegisterDataChannelMessageCallback()

void open3d::visualization::webrtc_server::WebRTCWindowSystem::RegisterDataChannelMessageCallback	(	const std::string &	class_name,
		const std::function< std::string(const std::string &)>	callback )

When the data channel receives a valid JSON string, the class_name property of the JSON object will be examined and the corresponding callback function will be called. The callback should return a string reply, which will be sent back to the client.

Parameters

class_name	The value of the `class_name` property of the JSON object.
callback	The callback function that will be called when a JSON object with the matching `class_name` is received via the data channel.

// Register callback in C++
auto ws =
visualization::webrtc_server::WebRTCWindowSystem::GetInstance();
ws->EnableWebRTC();
ws->RegisterDataChannelMessageCallback("webapp/input", "input",
    [](const std::string &data) {
        std::string reply = fmt::format("Received dataChannel message"
        " for class_name 'webapp/input' with data: {}", data);
        utility::LogInfo(reply);
        return reply;
    });

   /* Send message in JavaScript to trigger callback. this is
*  WebRTCStreamer object */
   this.dataChannel.send('{"class_name":"webapp/input",
       "data":"Test event"}');

◆ SendInitFrames()

void open3d::visualization::webrtc_server::WebRTCWindowSystem::SendInitFrames ( const std::string & window_uid )

Send initial frames. This flushes the WebRTC video stream. After the initial frames, new frames will only be sent at triggered events.

◆ StartWebRTCServer()

void open3d::visualization::webrtc_server::WebRTCWindowSystem::StartWebRTCServer ( )

Start WebRTC server in a background thread.

The documentation for this class was generated from the following files:

cpp/open3d/visualization/webrtc_server/WebRTCWindowSystem.h
cpp/open3d/visualization/webrtc_server/WebRTCWindowSystem.cpp

Data Structures

Public Member Functions

Static Public Member Functions

Additional Inherited Members