Using WebRTC to Create Multi-Party VR Experiences
by Jerod Venema, on October 29, 2018
The virtual reality industry has recently experienced a shift. What was once viewed as a predominately solo activity-centric experience, has become an avenue for developing rich social connections and collaboration. Developers have realized that virtual reality does not need to take place in isolation and are starting to look for ways to to increase social interaction in their applications.
But creating “social” virtual reality experiences is not simple. While there are server-based products on the market today that allow for integration of voice communication, multi-party collaborative live video and data has posed a significant hurdle for AR/VR application developers.
One of the most exciting possibilities for enabling multi-party virtual reality experiences is Web Real Time Communications (WebRTC). In this post we will explore how WebRTC can be used to make shared, cross-platform VR experiences possible and describe some benefits its integration bring to the AR/VR space.
Benefit #1: WebRTC Can Transmit Real-Time Video Between VR Headsets and Web Browsers
Web Real-Time Communications (WebRTC) is a standard that seeks to embed real-time voice and video natively between web browsers. With WebRTC, end-users are able to communicate in the web browser directly with each other via video without installing special plug-ins, extensions, or using a closed ecosystem communication vendor.
If you want users to transmit real-time video directly from an AR/VR enabled device to a browser without installing plugins or extensions, then WebRTC is currently your only option. In an excellent session at ClueCon earlier this year, Dan Jenkins, a WebRTC expert, demonstrated earlier this year that it is relatively easy to add WebRTC video conference streams into a virtual reality environment.
The LiveSwitch advantage: LiveSwitch supports peer-to-peer connections but moves far beyond basic WebRTC capabilities by also scaling for multi-party conversations, broadcasting, SIP-based telephony integration and more. Flexible architectures like the one used in LiveSwitch, reduce costs by choosing your application’s optimal topology for your virtual reality use case in the moment.To learn more about the importance of scalability, read our popular post how to scale your webrtc application.
Benefit #2: WebRTC for Low-Latency Audio and Video
Multi-party virtual reality experiences need to take place in real-time in order to mimic reality and maintain immersion. Having the lowest possible latency for audio and video transfer is key to building this experience. Latency can be affected by many factors including network characteristics, bandwidth availability and CPU usage. While other server products on the market today can also transmit voice, many of these options are much higher latency which can disrupt the user experience.
The LiveSwitch advantage: LiveSwitch achieves the lowest possible audio and video latency by providing full access to the complete media pipeline. This means that developers have the ability to customize and optimize the pipeline specifically for their use case (see chart below) to achieve even lower levels of latency than what is seen with traditional WebRTC, providing the most immersive experience possible for virtual reality end-users. Our professional services team has been able to achieve latency below 10 milliseconds with LiveSwitch by customizing the media pipeline for our customer’s specific use cases.
Benefit #3: WebRTC is Multi-Platform (Sort-Of)
According to Tsahi Levent-Levi, a WebRTC industry leader, WebRTC is supported (to varying degrees) in a number of different browsers and platforms including Chrome, Firefox, Safari, Edge, Android and iOS. This cross browser/platform compatibility makes it easier for virtual and augmented reality companies to create applications that function across web, mobile and desktop applications. However, this list is far from complete and does present some gaps in device usage.
The LiveSwitch advantage: LiveSwitch was designed to be one of the most versatile, real-time communications SDK and server stacks available on the market today. It has seamless interoperability and functionality across web, mobile and desktop environments. It supports a massive range of languages, platforms and browsers including Windows, Linux, macOS, iOS, Android, Java, Xamarin, Firefox, Explorer, Chrome, Safari, Opera and UWP making it one of the most flexible real-time voice and video products around. This cross-platform support makes it easier for developers to create applications for one platform and transfer that code to other platforms.
Benefit #4: Ultra Low Latency Datachannels Too
Sending and receiving data between two end-users can be a complicated process. However much of this headache can be alleviated by using WebRTC’s RTCDataChannel API to transfer data directly between peers.
The RTCDataChannel API supports a flexible set of data types which means that virtual reality developers can use them to deliver any type of data that they need to make multi-party VR experiences a reality. All of the properties necessary to build 3D voice and video such as coordinates, color, depth texture, 3D sound and mesh data can all be sent from one peer to another using one or more datachannels.
The LiveSwitch advantage: While WebRTC datachannels make it easy to transfer data from peer-to-peer, LiveSwitch takes it a step further with the ability to broadcast of data channels over MCU and SFU connections -- an industry first feature!
This means that you can send data to N people without taking on Nx bandwidth that is normally incurred in a simple peer-to-peer mesh network. In simpler terms, if 5 people were participating in a virtual hangout, and one participant needed to send everyone 3 Mbps of data, with LiveSwitch that person would only need to upload 3 Mbps of data in comparison to the 15 Mbps required for a peer-to-peer mesh virtual reality session. This number is unchanged regardless of the number of participants in a session. This ultimately reduces overall bandwidth utilization and improves headset battery power consumption which makes multi-party virtual reality that much more accessible for everyone.
Need to create immersive multi-party augmented or virtual reality experiences? We've got you covered. Check out LiveSwitch -- the massively flexible, highly scalable video conferencing solution.