Investigation into custom Unreal Engine streaming solutions

Posted by Luke Bermingham on 22 March 2021

Tags: Reports, Pixel Streaming, Unreal Engine, WebRTC

The contents of this report are Copyright © 2021 TensorWorks Pty Ltd and are licensed under a Creative Commons Attribution 4.0 International License.

Contents

• 1. Introduction
• 2. Background
  • 2.1 Streaming protocols
  • 2.2 Quality-of-experience techniques
  • 2.3 WebRTC forwarding technologies
• 3. Feasibility of various streaming solutions in Unreal Engine
  • UDP in the Unreal Engine
  • RTP in the Unreal Engine
  • RTP + RTCP in the Unreal Engine
  • RTSP in the Unreal Engine
  • RTMP in the Unreal Engine
  • SRT in the Unreal Engine
  • RIST in the Unreal Engine
  • Unreal Engine → WebRTC bridge → non-WebRTC receiver
• 4. Summary and recommendations
• Appendix A: Measuring latency
  • Embedded timecodes
  • Embedding metadata in the stream
  • Packet inspection in software
• Further reading
• References

1. Introduction

Since late 2018 ¹, the Unreal Engine has included a system called Pixel Streaming, whereby audio and visuals from a remote Unreal Engine instance are streamed to a WebRTC peer (typically, a web browser.) Standard inputs such as keyboard, mouse, and touch can then be transmitted back from the WebRTC peer to facilitate interaction with this Pixel Streaming Unreal Engine instance. ²

The Unreal Engine’s Pixel Streaming system is built upon WebRTC, which is well supported by modern web browsers. ³ WebRTC is specifically designed as a holistic solution for bidirectional, low latency real-time communication over the public internet. There are, however, a number of streaming scenarios that are either unidirectional, non-realtime, or operate in a controlled networking environment that does not suffer from the packet loss or the varying network conditions of the public internet. Examples of such streaming scenarios include: live broadcast, simulation software, legacy hardware, or simply streaming scenarios running in a completely controlled private network. These scenarios do not necessarily require, nor support, the conglomerate of technologies that are WebRTC.

If these aforementioned scenarios are unable to support a WebRTC connection it means that the Unreal Engine Pixel Streaming system cannot be used directly. This is unfortunate, as streaming of the Unreal Engine’s audio, visuals, and inputs in these use cases is, we argue, highly desirable. Fortunately, many of these streaming use cases, particularly blackbox hardware and simulation software, already support receiving streaming data using existing streaming protocols such as RTP or RTMP. The support for non-WebRTC streaming protocols in these use cases presents two potential options:

1. Implement these non-WebRTC streaming protocols inside the Unreal Engine Pixel Streaming system and stream directly from the Unreal Engine to a receiver using these protocols.

2. Use the existing WebRTC-backed Pixel Streaming system to stream to an intermediate WebRTC gateway server that effectively forwards the WebRTC media streams via a supported streaming protocol such as RTP or RTMP.

The purpose of this report is to investigate each option. Specifically, we will identify candidate approaches for these options and then evaluate each approach in terms of its advantages, limitations, and general feasibility. We conclude this report with our recommendations regarding Pixel Streaming to these non-WebRTC receivers.

2. Background

In this section we present an overview of the relevant streaming protocols, various quality-of-experience techniques that generally accompany real-time streaming protocols, and a review of some WebRTC gateway servers and forwarders. This overview provides the relevant context to inform our investigation into the advantages, limitations, and feasibility of each potential solution in Section 3.

2.1 Streaming protocols

Streaming of real-time data, such as video, audio, and control input, is a task that has been attempted and improved upon many times in the past. As part of this report we have conducted an analysis of some of the previous works that we consider relevant as potential streaming protocols for our purpose. The overview table below provides a brief description and the key details about the relevant protocols.

2.2 Quality-of-experience techniques

We argue RTP is likely the simplest protocol to implement of any listed in Table 1. However, unlike SRT, RIST, RTMP, and WebRTC, the RTP protocol has no in-built mechanism to handle packet loss or network jitter - these challenges are deferred to the application developer. Thus, it seems prudent to discuss how these challenges would be addressed if, say, a pure RTP streaming solution inside of Unreal Engine were to be implemented.

Fortunately for the developer, there exists a number of well known and effective techniques to address packet loss and network jitter of streaming data. We call such approaches “quality-of-experience” (QoE) techniques. Some known QoE techniques include forward error correction (FEC), automatic repeat request (ARQ), jitter buffering, and packet loss concealment. We briefly introduce and explain each of these QoE techniques below. Additionally, we highlight that all proper WebRTC implementations will implicitly include FEC and jitter buffering QoE techniques. ¹⁵ Furthermore, Google’s WebRTC, the implementation that Unreal Engine’s Pixel Streaming system uses, also includes a number of custom packet loss concealment techniques. ¹⁶

Forward Error Correction (FEC)

Forward Error Correction (FEC) is the process of transmitting redundant information alongside streaming data so that when a packet is lost this redundant information can be used as an error correcting code to recover the lost information. ¹⁷ In practice, this requires keeping a buffer of received packets so that when missing packets are detected future packets can be used to reconstruct the lost information without asking for a retransmission of old data. ¹⁸ The tradeoff is these redundant error correcting codes increase the total transmission size. However, it should be noted that the more redundancy that is added the better FEC works in practice. ¹⁸ The amount of redundancy that is acceptable is largely defined by the bandwidth capabilities of the underlying network. As an indicator for the sort of loss recovery one could expect, a real-world study ¹⁸ demonstrated that with a 50% bandwidth overhead an RTP-based FEC implementation could recover streaming data in scenarios where packet loss was approximately 5 - 10%.

Automatic Repeat Request (ARQ)

Automatic Repeat Request (ARQ) is a technique where the receiver requests for the sender to retransmit a packet if it is lost. ¹⁹ ARQ requires the introduction of a buffer on the receiving end to determine if packets are truly lost or simply out-of-order. ¹⁹ ARQ has two major disadvantages compared to FEC:

1. ARQ requires bidirectional communication between sender and receiver. ¹⁹
2. ARQ introduces additional latency waiting for retransmission to occur. ¹⁹

However, unlike FEC, ARQ does not require a great deal of additional transmission bandwidth. Additionally, ARQ is able to recover from large, contiguous, runs of lost packets, which is a scenario FEC would struggle with. ²⁰ We highlight that Haivision, the creators of SRT, use ARQ in their streaming protocol because “it requires less bandwidth than other error correction methods… and introduces only a small degree of latency to the transmission”. ²¹

Jitter Buffering

Jitter buffering is simply the process of maintaining a buffer on the receiving end to store incoming packets. ²² It is used for packet reordering when packets arrive out-of-order and also to provide a constant playback stream of the receiving end. ²² The disadvantage of jitter buffering is that latency is increased directly by the size of the buffer. While fixed size buffers are simpler to implement, it is also possible (and desirable) to dynamically resize the jitter buffer based on recent network conditions. Dynamic jitter buffers can use stream statistics from protocols such as RTCP to decide when to resize the buffer. We highlight that FEC and ARQ both contain jitter buffering as part of their practical implementations.

Packet Loss Concealment

Packet loss concealment is actually a family of techniques whose purpose is to decide how to progress in the presence of a missing packet. Unlike FEC, ARQ, and jitter buffering, which are general QoE techniques for any streaming data, packet concealment techniques are typically designed to specifically handle lost audio or video packets by exploiting the properties of audio and video data. For example, in some audio streams when packets are lost a technique called waveform substitution is used, whereby some already received audio packets are carefully reused and stitched into the received audio to extend the current voice sound until new data is received. ²³ Another example is Google Duo, which uses the WaveNetEQ generative model to interpolate parts of speech which are missing due to packet loss. ²⁴

2.3 WebRTC forwarding technologies

As we mentioned in Section 1, the other option, besides implementing alternate streaming protocols inside of Unreal Engine, is to stream using Unreal Engine’s existing WebRTC-based pixel streaming to a WebRTC gateway/forwarding server. This gateway/forwarding server would then translate the WebRTC stream to a protocol that the receiving end can understand, such as RTP, then finally, the gateway/forwarding server will transmit the stream along to the receiver. Fortunately, due the wide adoption of WebRTC there are existing gateways/forwarding servers that may be suitable for this purpose. We present some of the relevant options below.

Janus

Janus is a general purpose WebRTC server written in C. It is designed such that the base Janus application has a small footprint and most extra functionality is achieved through plugins. ²⁵ For the purposes of forwarding WebRTC media through other protocols, Janus provides a “NoSIP” plugin that acts as an “RTP bridge” that relays media from a WebRTC peer to a receiver using RTP/RTCP. ²⁶ There are also the Janus plugins “Videoroom” ²⁷ and “Janus-RTP-Forward-Plugin” ²⁸ written by the community, which also appear to be viable options. We highlight that Janus is open source and licensed under GPL 3.0. While GPL 3.0 is incompatible for linking into the Unreal Engine this is likely a non-issue because a Janus WebRTC forwarding server would sit completely outside the Unreal Engine.

Pion

Pion is a WebRTC implementation written in Go and unlike Google’s WebRTC, Pion is specifically designed to be fast to build and customise. ²⁹ While Pion is not specifically a WebRTC gateway or server it does contain an “RTP-Forwarder” example that illustrates how to use it as a WebRTC peer that forwards RTP packets elsewhere. ³⁰ In terms of implementation, we recognise that Go is not trivially callable from Unreal Engine C++; however, this should not be a problem because Pion would be deployed outside of Unreal Engine and communicate with Pixel Streaming as a standard WebRTC peer.

Other options

We also investigated GStreamer WebRTC ³¹ and Kurento WebRTC server ³², both of which seemed suitable, but somewhat less documented than Janus or Pion regarding WebRTC to non-WebRTC stream forwarding. Additionally, it also seems possible to implement a WebRTC forwarding server using Google’s WebRTC; however, we would not advise this course of action, because in our experience, Google’s WebRTC is complex, time consuming to build, and non-trivial to update.

3. Feasibility of various streaming solutions in Unreal Engine

In this section we evaluate the feasibility of various streaming solutions in terms of their implementation into the Unreal Engine, their general advantages and limitations, and we then briefly present our conjecture on how each solution would fare in an unreliable network.

Specifically, we will explore the feasibility of each protocol by discussing each of the following criteria:

1. How to implement. We briefly describe the process of implementing a potential solution with the Unreal Engine’s Pixel Streaming system. Are there any Unreal Engine compatible APIs to utilise or study as a suitable reference implementation?

2. Advantages. What are the advantages of implementing this solution over other options?

3. Limitations. What are the limitations of implementing this solution?

4. Unreliable network use case. How would we expect this solution to fare in a real network with packet loss, jitter, and changing network conditions (e.g. the public internet)?

UDP in the Unreal Engine

UDP streaming is likely the simplest option to implement inside of the Unreal Engine; however, it has several limitations that, we argue, make it an untenable choice. Implementing video and audio streaming in the Unreal Engine using UDP will not require the implementation of any specific protocol or library as the Unreal Engine already supports transmission of UDP packets. ³³ Additionally, transmission of data using UDP will lack the packet overhead that other transmission protocols require. However, pure UDP, is very limited; it has no support for packet loss, out-of-order delivery, or jitter. For example, even in private, reliable networks, UDP will potentially drop packets or have some out of order delivery. ³⁴ For example, when we consider the UDP transmission of H.264 encoded video in a unreliable network, the loss of packets may well result in the video stream becoming unrecoverable. ³⁵ It is our view that these limitations make pure UDP an untenable choice. However, there are several additional techniques that can be implemented on top of a UDP streaming solution, such as FEC and jitter buffering, that will mitigate these limitations to some extent. However, we would argue that if mitigating techniques are being used to overcome the intrinsic properties of UDP then other protocols, such as RTP, already exist, and due to their wide adoption, should be preferred.

RTP in the Unreal Engine

RTP is a lightweight protocol that can be implemented in the Unreal Engine using the inbuilt UDP transmission APIs. If a reference implementation is required we recommend studying uvgRTP ³⁶³⁷ and Live555 ³⁸. The major feature of RTP, compared to pure UDP, is that RTP transmits sequence information in its packets to aid in reconstruction when out-of-order delivery occurs. However, RTP itself has no inbuilt QoE techniques to actually address out-of-order delivery, which means if QoE techniques such as jitter buffering or FEC are required, these QoE techniques must then be implemented by the developer. In a reliable network without extra QoE techniques implemented, we would expect RTP’s effectiveness is mostly similar to UDP. However, with QoE techniques implemented, we expect RTP to operate effectively in a reliable network. Additionally, in a reliable network, jitter buffers could be kept relatively small, thus minimising latency. However, in an unreliable network, RTP would require QoE techniques to deliver a functional real-time streaming experience. However, we highlight that the QoE techniques commonly paired with RTP, FEC and jitter buffers, will introduce tradeoffs in bandwidth and latency, respectively, when operating in an unreliable network.

RTP + RTCP in the Unreal Engine

RTCP is designed to monitor real-time streams and calculate statistics, which are then used to dynamically modify QoE techniques or dynamically adjust the source stream. Should the implementation of RTCP into the Unreal Engine be pursued, it is our recommendation to study the existing RTCP library in Live555. ³⁸³⁹ We highlight that to make full use of RTCP it would have to be implemented in the receiving end too, which may not be possible in some blackbox systems. However, assuming RTCP is implemented on both ends and RTP is using jitter buffering, we expect RTCP would greatly improve the quality of experience in unreliable networks by allowing the jitter buffers to dynamically resize based on network conditions. Contrasting this to a pure RTP jitter buffer that has to assume a fixed size, we would expect the RTP + RTCP solution to have reduced latency as network conditions recover. Additionally, even if jitter buffering is not used, we would still expect RTP + RTCP to produce a better experience than pure RTP in unreliable networks because the source quality of the stream can be dynamically adjusted using information from RTCP.

RTSP in the Unreal Engine

RTSP is a protocol for controlling streaming media servers. ⁶ In practice, it is typically a thin protocol that uses RTP and RTCP for the underlying media streams and control messages. Therefore, we consider the practical advantages and disadvantages of implementing RTSP in the Unreal Engine to be much the same as an RTP + RTCP solution. Thus, we refer the reader to our above remarks regarding RTP + RTCP. Additionally, when seeking a reference implementation for RTSP we recommend studying Live555 ³⁸.

RTMP in the Unreal Engine

RTMP is a widely supported TCP-based protocol that was developed by Macromedia/Adobe ⁷. In terms of implementing RTMP into the Unreal Engine, audio and video streams from the Unreal Engine’s Pixel Streaming system would have to be chunked and transmitted using the Unreal Engine’s TCP API ³³ in the RTMP packet format. We recommend studying iReader’s media server for an existing RTMP implementation. ⁴⁰ The major disadvantage of RTMP is that it uses TCP as its underlying transport, which in an unreliable network will cause latency to bloat as redelivery attempts occur. ⁴¹

SRT in the Unreal Engine

SRT is a low latency UDP-based streaming protocol created by Haivision. ¹⁰ SRT is open source and Haivision provides a production ready SRT library for developers. ¹¹ To implement SRT inside the Unreal Engine’s Pixel Streaming system we would recommend incorporating Haivision’s SRT library into its own plugin or modifying the existing Pixel Streaming plugin to include the SRT library. The advantage of SRT is that it is specifically designed for low latency streaming across unreliable networks and unlike other UDP-based protocols, such as RTP, SRT itself contains a mechanism to handle packet loss via ARQ. The main disadvantage we foresee with a solution that uses SRT is that SRT is relatively new and is unlikely to be supported by many devices on the receiving end.

RIST in the Unreal Engine

RIST is very similar to SRT, it too is a low latency UDP-based streaming protocol. ¹³ Unlike SRT, which was created by a private corporation, RIST was created by the Video Services Group, a non-profit “comprised of service providers, users and manufacturers dedicated to interoperability, quality metrics and education for media networking technologies”. ⁴² To implement RIST we recommend incorporating librist ¹², the official RIST library, into its own plugin or including it in the existing Pixel Streaming plugin. Much like SRT, the main disadvantage of implementing a solution that uses RIST is that it is relatively new and unlikely to be supported by many devices on the receiving end.

Unreal Engine → WebRTC bridge → non-WebRTC receiver

This solution involves leveraging the existing Pixel Streaming system in the Unreal Engine to stream to a custom WebRTC bridge that then forwards the received video and audio from the Unreal Engine to some arbitrary non-WebRTC receiver using a widely support streaming protocol such as RTP, RTSP, or RTMP. Existing WebRTC restreaming servers include Janus ²⁵ and Wowza streaming engine ⁴³; however, the Go-based WebRTC library, Pion ²⁹, is also an option if there are specific requirements beyond simply forwarding streams.

The advantage of this solution is that it requires no modification to the Unreal Engine or the Pixel Streaming plugin and moves the requirement to stream to other protocols outside of the Unreal Engine. Additionally, the deployment of a WebRTC bridge, particularly on its own hardware, is more scalable than streaming directly from the Unreal Engine while also eliminating the possibility of resource competition with the Unreal Engine Pixel Streaming application. Furthermore, due to WebRTC being the underlying streaming technology, this approach inherits all of the low latency, high resiliency, properties of WebRTC if configured properly.

Conversely, the disadvantage of this solution is that it requires another component, potentially running on its own hardware. Additionally, this approach may introduce some latency as the stream is forwarded through the WebRTC bridge. Furthermore, if the WebRTC bridge is not in the same local network as the receiver this approach will inherit the same qualities as whichever protocol is selected to stream from the WebRTC bridge to the non-WebRTC receiver.

4. Summary and recommendations

Based on Section 3 there are some solutions we can disregard from our recommendations as they do not satisfy our goal of ensuring compatibility with legacy and/or blackbox streaming clients. Specifically, SRT and RIST should be disregarded because they are likely to be incompatible with most legacy streaming receivers. Furthermore, it is not clear whether RTSP has any benefits over simply implementing RTP and RTCP in an Unreal Engine Pixel Streaming scenario; therefore, it too can be disregarded. Lastly, a pure UDP streaming solution may suffer from packet loss issues, even in a private network, so we will also disregard it too.

Thus we consider RTP, RTP + RTCP, RTMP, or a WebRTC forwarding bridge to be the most feasible solutions to support streaming from the Unreal Engine’s Pixel Streaming system to a non-WebRTC receiver. However, of these four options, a WebRTC forwarding bridge is our recommended solution.

We argue that the RTP, RTP + RTCP, and RTMP solutions all have one major disadvantage compared to a WebRTC forwarding bridge. That is, if stability in poor or varying networking conditions is required then these protocols do not provide a solution on their own. RTP and RTP + RTCP can be implemented with FEC, packet loss concealment, and jitter buffering to address these issues; however, this must be implemented, to some extent, on both ends, which is a non-trivial task and may be an infeasible task if the receiving end is a blackbox or unmodifiable legacy system. Whereas, a WebRTC forwarding bridge gains all the low latency and poor networking resilience that are inherent to WebRTC. We highlight that to utilise these benefits of WebRTC to maximum effect we recommend installing the WebRTC forwarding bridge in the same local network as the non-WebRTC receiver. If the WebRTC forwarding bridge is not in the same network as the non-WebRTC receiver then this solution will be subject to the same disadvantages as the underlying forwarding protocol (e.g. RTP, RTMP).

Furthermore, even in a scenario where stability in poor or varying network conditions is of no concern, we still recommend the WebRTC forwarding bridge. The WebRTC forwarding bridge is the only solution that does not require making any modifications to the Unreal Engine. This mitigates several issues, including: the maintenance cost of updating the solutions as changes occur in the Unreal Engine source code, licensing conflicts with the Unreal Engine, and the complexity of integrating third party libraries inside the Unreal Engine. Additionally, it is the only solution that operates completely outside the Unreal Engine and is therefore far more scalable.

To demonstrate the viability of this solution we have provided a proof-of-concept using Pion WebRTC to forward RTP streams from the Unreal Engine’s Pixel Streaming system to a non-WebRTC receiver (FFplay in this case). This proof of concept can be found at: https://github.com/TensorWorks/Unreal-PixelStreaming-RTP-Bridge

Appendix A: Measuring latency

In any streaming solution measuring the overall and per-component latency is useful to debug, optimise, and configure the system. Based on our examination, it appears that there is no embedded mechanism within the streaming protocols we have analysed for this purpose. In this appendix we present a number of techniques of varying implementation complexity and accuracy for the reader’s consideration. We restrict our techniques to video streaming scenarios.

Embedded timecodes

This technique requires timecodes are rendered into the source video. This video is then transmitted and the received video image containing the rendered timecode can be visually inspected at each component and compared with the source video timecode. The difference in timecodes gives a rough estimate of the latency between the source and the tested component. The major problem with this technique is synchronising the manual inspection. While this technique is inaccurate, it is simple to implement.

Embedding metadata in the stream

This technique requires timestamps to be transmitted inside the video stream by using the existing metadata fields that are supported by the underlying media. For example, H.264 supports a “user data unregistered SEI message” ⁴⁴ that others have successfully used for this purpose. ⁴⁵⁴⁶ We highlight that this technique requires components in the streaming system to have synchronised clocks. Once the timestamp is received it is compared with the system clock to determine the latency. The advantage of this method is that it is highly accurate. The disadvantage of this approach is it artificially inflates the bandwidth, which may itself impact latency.

Packet inspection in software

This technique does not require any modification to the streaming system, instead it requires that packet inspection software is installed on each component in the streaming solution. When the latency test begins the packet inspection software starts collecting all the relevant packets and then logging the packet contents and the time of receipt to a file. Once the latency test is complete all logged data across all components is processed offline such that a given packet will have a corresponding timestamp at each component that received or transmitted it. Using these timestamps, overall and per-component latency can be calculated over time. We highlight one major disadvantage of this technique is that it is only possible when each packet can be uniquely identified, which is possible in protocols such as RTP where each packet is given a unique sequence number. However, in configurations where packets are encrypted or streams are interleaved together this technique becomes challenging to implement. However, if this technique is pursued, we recommend using the packet inspection software Wireshark ⁴⁷ to collect the relevant packets. In addition to packet collection, Wireshark also has an RTP stream analysis feature ⁴⁸⁴⁹ that reports packet loss and jitter, both of which are useful complementary sources of information when analysing stream latency.

Further reading

• The textbook, “RTP: Audio and Video for the Internet” is a detailed textbook about the RTP protocol and provides an in-depth discussion about all the considerations one would have to make to realistically use RTP for streaming audio and video on the public internet. For example, the book has chapters dedicated to packet loss concealment and how to add security on top of RTP. ⁵⁰

• There exists an RFC for a proposed standard regarding an “RTP Payload Format for H.264 Video”. According to the RFC, “the RTP payload format allows for packetization of one or more Network Abstraction Layer Units (NALUs), produced by a H.264 video encoder, in each RTP payload”. While this RFC is only a proposed standard and is not yet accepted (opened in 2011) it is probably best to follow for future-proofing purposes if streaming of H.264 via RTP inside the Unreal Engine were to be attempted. ⁵¹

• This article ⁵² discusses how one might implement QoE techniques such as jitter buffers on top of RTP.

• Haivision’s SRT whitepaper describes, at a high level, how SRT is different from other real-time media streaming protocols and how it handles poor network conditions at the protocol level. ⁵³

• In this blog post ⁵⁴ from Haivision, the creators of SRT, they provide some history on SRT and how they came up with their solution based on their work with live broadcast transmission. They mention their previous approach of using MPEG transport streams on protected networks, which when packet loss is low, works perfectly, they claim. They go on to mention that if packet loss became a problem on those protected networks they used Forward Error Correction (FEC), which solved their packet loss problems entirely. It was only when their customers wanted to move away from protected networks and broadcast between sites that they required a solution for low-latency, error-tolerant, transmission over the public internet. ⁵⁴

References