Enabling higher quality Pixel Streaming experiences with AV1

Posted by William Belcher on 5 August 2024

Tags: Pixel Streaming, Unreal Engine

The needs of the Internet have shifted. Interactive video applications such as telemedicine, video conferencing and cloud gaming have seen surges in popularity over recent years. However, the default video compression scheme used in Pixel Streaming, H.264, struggles to provide high quality video in these low latency, low bitrate, interactive video scenarios. Fortunately, the Alliance for Open Media (AOMedia) recognised these shortfalls and designed AOMedia Video 1 (AV1), an open, royalty-free video coding format designed for video transmission over the Internet.

Whilst partaking in a company-wide R&D jam, TensorWorks identified the lack of support for hardware encoding with AV1 in Unreal Engine. This blog post will explain the benefits of AV1 and how you can leverage it for your Pixel Streaming application today with the release of UE 5.4.

Contents

• Introduction to AV1
• Performance
• Requirements
• Summary

Introduction to AV1

State of the art video codecs such as AV1 can dramatically improve video call quality, while using similar bandwidth to existing standards such as H.264. These benefits, however, aren’t free. Encoding video with AV1 demands greater amounts of computation resulting in longer encoding times. This drawback has meant that despite mass migration to AV1 for video-on-demand encoding, adoption of AV1 in live-streaming applications has been slow.

However, significant progress came in late 2022 when Nvidia and Intel announced support for real-time AV1 encoding on Nvidia’s Ada Lovelace and Intel’s Arc Alchemist GPUs. As Unreal Engine lacks support for hardware accelerated video encoding on Intel GPUs, the remainder of this blog post will focus on the performance observed on Nvidia hardware.

Nvidia claims that AV1 encoding using the Nvidia Encoder (NvEnc) is able to achieve a visual quality approximately 10% better than H.264 during low bitrate scenarios. In fact, in their blog post, they claim that the visual quality of AV1 outperforms H.264 across the entire range of bit rates they tested! Let’s see how AV1 fares in our testing.

Performance

TensorWorks tested the performance of AV1 and H.264 at varying bitrates to assess each codec’s VMAF score. The results can be seen below:

The results show an approximately 35-48% improvement in VMAF score when using Pixel Streaming’s encoding settings at these bitrates. While these improvements are impressive, what we really want to know is, are the visuals observably better? For this, we need to record some Pixel Streaming output from H.264 and AV1. Let’s see how they visually compare!

Video comparisons can be seen in the table below (the low bitrates show the biggest improvements):

Requirements

Support for AV1 in AVCodecs and Pixel Streaming has landed in UE5.4!

Following the TensorWorks’ jam, we set out to add AV1 support in time for UE 5.4. As part of our collaboration with Epic, we were able to add NVIDIA AV1 encoding support for all Unreal Engine licensees with the functionality landing in this commit (Note: you will need to be a member of Epic’s Github org to see this commit).

To leverage the benefits of AV1 with Pixel Streaming, you’ll need to ensure that your local machine (or cloud instance such as GCP’s G2) has:

• A compatible Ada Lovelace or greater, generation GPU and
• That your driver version is 555.25 and greater for Windows, or 520.56.06 and greater for Linux.

And that’s it! You can now append -PixelStreamingEncoderCodec=AV1 to your Unreal Engine application’s existing launch arguments and start streaming.

To check that your application is streaming AV1, you can consult the statistics in the information panel once a stream has started. The info panel can be opened by clicking the (i) on the left side of the browser. The video codec will be specified on the right.

Summary

AV1 encoding for Pixel Streaming is now available in the 5.4 release of the Unreal Engine. Empirical tests observed that AV1 is able to retain higher fidelity information at low bitrates which directly translates to a significantly higher VMAF score.

As with all of the latest technology, there comes a financial penalty for being an early adopter. As of writing (August 2024), a basic GCP compute instance supporting only H.264 (Nvidia T4) will cost at least $0.35USD/hr, whereas an instance with a GPU supporting AV1 (Nvidia L4) will cost at least $0.71USD/hr. In conclusion, AV1 with Pixel Streaming is here and it’s impressive. Its ability to retain visual information in low bitrate scenarios sees it becoming another key tool that we have in the pursuit of higher quality streaming experiences. While the increased cost of compute instances will slow its adoption, we expect this price discrepancy to reduce as we see more cloud providers pick up AV1 capable GPUs.