High Fidelity Web Client

The janusvr team built a web client that communicated with the HiFi servers and blended functionality with the native client.

Originally published January 2019

tags: devlog hifi janus vrba

Preliminary efforts to bring High Fidelity functionality to the web.

Developer Notes

This project consisted of two parallel development projects - the HifiWeb client, and a webrtc-to-hifi relay server. Both projects required us to quickly obtain familiarity with High Fidelity's underlying network protocol, which turned out to be a larger undertaking than we initially thought.

The bulk of the work involved in creating the client once the relay was functional consisted of writing a binary protocol handler in JavaScript to handle HiFi's network packets. We started off implementing the packets necessary for negotiating the connection handshake process. Once we finished that we started implementing more packets - AvatarData, SelectedAudioFormat, MixedAudio, etc. It took us about a month to get the first working session where a HiFi user was able to tell that a HifiWeb user was in the same room, and they could see each other moving about.

First Test

November 19th, 2018

Connecting to a High Fidelity domain with both a HiFi client and a JanusWeb client. Both clients can see each other's positions in the same virtual space, and avatar updates are being sent back and forth.

From there we started implementing VOIP. Playing back HiFi's MixedAudio packets turned out not to be too hard, but the voice input side of things caused us some problems due to browser inconsistencies with the WebAudio API. The solution we ended up with uses an AudioWorklet to handle the audio on a separate thread, which leads to better performance when the app is busy doing other things like rendering the world. We ran into some issues with third party audio resampling libraries introducing sound artifacts, but once we realized the problem and got all the audio parameters dialed in, the resulting audio finally came through crystal clear.

In the course of developing the relay and the client packet handling library, we developed some debug tools to help us along the way. A packet visualizer lets us see the flow of packets that are sent between the client and the server. We can drill down to conversations between the client and each assignment-client node, and inspect the order and contents of individual packets as they're exchanged. This proved to be invaluable for debugging connection issues and implementing parsers for the various types of packets High Fidelity supports.

How's it work?

Where does one begin the path of building a web client to a native metaverse frontend? The goal is to see and hear eachother, so first we begin with the space.

Next, we add the communication channel. The WebRTC Relay is a server-side console application that bridges WebRTC clients with HiFi servers.

The relay connects to the web client via a WebRTC peer connection and then a single data channel is established for sending/receiving packets to/from the web client. We decipher which node to send packets to/from by encapsulating packets with the node type we are communicating with (done on both the relay and client side). The relay can make connections to multiple web clients. There is also initial work allowing users to log in and authenticate via the metaverse API.

In our initial implementation, we dug into HiFi source code in order to connect to the servers (including making requests to the Stun and ICE servers, domain ID lookup via the metaverse API given a domain name, making the initial domain connect request to the domain server, and then handling creating UDP connections to the different nodes we receive from DomainList packets). Our initial implementation had the relay handle handshaking, obfuscation, and hashing packets; a lot of that packet handling has been moved to the HiFi web client. Some parts of the protocol are still kept on the relay side, such as contacting the Stun/ICE servers, and domain connect requests (where some packets include the relay's IP and hardware address).

TODO: We have the relay compiling on OSX and Linux, but have yet to compile librtcdcpp (a library dependency for WebRTC) for Windows:

https://github.com/chadnickbok/librtcdcpp

Merging

Detecting avatars between web and native client

Final Presentation

History

Janusweb running as a High Fidelity tablet app (June 11, 2018)

JanusWeb / High Fidelity Integration (Jun 16, 2018)