MPEG-5 LCEVC in remote desktop streaming

Introduction

In our previous blog post, we explored the benefits of the latest standard by MPEG and ISO, MPEG-5 Part 2 LCEVC (Low Complexity Enhancement Video Coding), for the desktop virtualization use case. We demonstrated how this innovative codec enhances video compression, resulting in improved performance and reduced bandwidth usage.

Our recent advancements have built on this foundation. By integrating LCEVC with FreeRDP, an open-source implementation of the Remote Desktop Protocol (RDP), we demonstrate the enhancement layer's integration capabilities on an infrastructure representative of that used by remote desktop services (DaaS/VDI) and application (RDA) providers, leveraging existing hardware encoders and decoders. This means the improvements can be readily adopted without needing infrastructure changes, making the technology accessible and practical for widespread use.

In this post, we will discuss these advancements in detail, showcasing the integration process, the performance improvements, and the availability of a demo for users to experience the benefits firsthand.

TL;DR: Enhancing remote desktop performance with an enhancement layer

• Bitrate reduction: achieved a 25% reduction, enhancing bandwidth efficiency while keeping video quality.

• Encoding time: reduced by over 33%, optimizing computational resources.

• Backward compatibility: the enhancement layer demonstrated seamless integration with existing codecs and protocols, facilitating the adoption of performance improvements on current hardware without substantial changes.

These advancements demonstrate substantial improvements in remote desktop technologies, making them more efficient and environmentally sustainable while reducing total ownership costs.

Challenges and innovations in remote desktop experiences

As remote work becomes the norm, IT infrastructure faces significant strains. Increased demands for remote desktop solutions have escalated operational costs due to the need for high computational power and bandwidth. This is especially true as industries like graphic design, video editing, or medical imaging require streaming at ultra-high resolutions, up to 8K.

Bandwidth limitations present a major challenge, where traditional video codecs often fail to provide acceptable quality at lower bitrates, impacting user experience and productivity. Additionally, running remote desktop servers requires robust hardware to manage real-time video encoding and decoding, leading to higher energy consumption and operational costs.

Despite the availability of advanced video codecs like VVC and AV1, their hardware acceleration adoption is slow. Many organizations are slow to upgrade their hardware infrastructure due to the associated costs and complexity. This hinders the widespread use of these technologies, limiting their real-world benefits. Enhancement layers like MPEG-5 Part 2 LCEVC provide a solution by preprocessing videos to enhance quality and reduce size before encoding with base codecs like AVC or HEVC. This method reduces bandwidth and ownership costs while maintaining video quality.

While the transition to advanced codecs has been slow due to high costs and complexity, the introduction of such enhancement layer technologies underscores a shift towards more sustainable and efficient remote work environments.

Our approach

In previous experiments, we tested MPEG-5 Part 2 LCEVC to validate its claim for the desktop virtualization use case, achieving over a 20% BD-Rate reduction for H.264 as the base codec. The evaluation was conducted in a controlled codec environment using a video dataset containing remote desktop and screen content for testing.

Moving forward on our R&D roadmap, our focus now shifts to integrating the enhancement layer into a communication protocol tailored for remote desktop applications. This Proof of Concept (PoC)  is designed to demonstrate the advantages of this integration in a scenario closely mirroring real-world conditions, especially those encountered by our DaaS/VDI and Thin Client customers.

Given that a large portion of our Thin Client customers operate on Linux systems, while the remote desktop market predominantly utilizes Windows, our attention is directed towards the Remote Desktop Protocol, RDP, which offers multiple implementations compatible with both operating systems. Among these implementations, FreeRDP stands out as it aligns with all our requirements: it is open source, widely adopted, and supported by an active community.

In order to ensure that our research accurately mirrors real-world usage scenarios, we have implemented a continuous monitoring framework. It captures and encodes live desktop activities, offering real-time data for performance assessment on the host desktop where it is installed. Similarly, a remote desktop server, like ours featuring LCEVC with FreeRDP, allows users to engage with the server via the RDP client, replicating typical user interactions. Unlike relying solely on selected dataset video inputs, this live setup enables us to gather essential encoding performance metrics directly from real-world usage scenarios, enhancing our capability to evaluate and enhance system performance comprehensively.

It is important to note the complexities involved in monitoring the effectiveness of encoding enhancements directly on the RDP stream due to the challenges of integrating extensive monitoring logic directly into the FreeRDP code. In addition, continuous monitoring and objective quality metrics must be maintained without impacting the RDP application. To address these challenges, we opted for external monitoring of RDP.

Through this approach, users can leverage our monitoring system while the LCEVC with FreeRDP demo client interacts with the served desktop. Thus, users can observe the immediate impact of various encoding configurations on the same desktop through our monitoring dashboard. This setup is crucial for evaluating the enhancements in video quality and encoding efficiency, providing a transparent and interactive way for users to see the benefits of our technology in action.

Design of demonstrators

LCEVC within FreeRDP

Integrating LCEVC with FreeRDP aims to enhance the standard H.264 video codec in the remote desktop protocol. This integration adds an enhancement layer to improve video quality and compression and reduce encoding complexity. The design includes:

• Base codec (H.264): utilizes H.264 for initial video encoding.

• LCEVC enhancement layer: applies LCEVC technology from V-Nova’s encoder SDK to enhance video quality above the base H.264 layer.

• Integration into FreeRDP: implements the VNova’s LCEVC encoder SDK as a subsystem within FreeRDP, interfacing with its video processing components.

• Configuration for low latency: optimizes the LCEVC encoder settings to minimize encoding latency, which is essential for real-time interactions.

• FreeRDP adjustments: modifies FreeRDP’s logic to handle the enhanced video stream efficiently, ensuring optimal responsiveness for the RDP client.

Key challenges addressed included SDK integration, latency optimization, and adjustments in FreeRDP. Emphasis was placed on seamless protocol integration, cross-platform testing, and enhancing the user experience through meticulous evaluation of video quality, responsiveness, and overall usability.

Continuous monitoring and KPI gathering

A monitoring framework that utilizes live desktop screen recordings, dual encoding enhanced with and without enhancement layer, real-time data capture, and the collection of key performance indicators (KPIs) like bitrate, objective video quality, and encoding time.

• Desktop as Input Source: captures real-time desktop activities for encoding.

• Dual encoding: compares the performance of LCEVC enhancement on top of x264 against standalone x264 encoding. This setup allows for simultaneous encoding using both methods, enabling direct evaluation of LCEVC's efficiency against the same live desktop content.

• KPIs Collection: records vital data, including bitrate and encoding duration. Videos are encoded using standard H.264 and enhanced with LCEVC.

• Quality Metrics: computes and logs PSNR and VMAF scores to assess video output quality objectively.

• Data Integration: formats logs for compatibility with InfluxDB.

• Live Statistics Board: transmits data to InfluxDB, enabling real-time analysis through a Grafana dashboard, which facilitates monitoring and comparing each encoding method’s impact.

This comprehensive monitoring framework aims to continuously refine design choices, ensuring optimal video quality while minimizing bitrate and encoding time. It is tailored to diverse use cases and maintains high user experience standards across remote desktop environments.

What we want to achieve and measure

• Customization and optimization opportunities: Optimizing the enhancement layer and the encoder simultaneously leaves room for customization and design decisions. In the end, we set our priority on keeping the video quality constant and reducing bitrate and encoding time. Still, other priorities are also possible and desirable in specific use cases.

• Latency considerations in remote desktop environments: minimizing latency is essential for user satisfaction, ensuring that encoding does not introduce substantial delays.

• Reducing total cost of ownership (TCO): integrating LCEVC on top of H.264 can bypass the need for more resource-intensive codecs like H.265. This achieves comparable or superior BD-Rate efficiency with lower encoding times and broader compatibility, thus minimizing infrastructure costs.

Results

We successfully integrated version 3.10 of the V-Nova LCEVC encoder SDK into FreeRDP version 3.6.3, achieving an RDP stream of 25 fps with minimal delay when using x264 as the base codec. The successful integration has allowed us to expand the MPEG part 2 technology's horizons for DaaS and Thin Client implementations, laying a solid foundation for future improvements. Initial user feedback on Quality of Experience (QoE) has been positive, validating the prototype’s strong performance and setting a reliable standard for future improvements.

Figure 1 shows the results of monitoring a screen while playing different videos under typical DaaS and Thin Client usage situations (Figure 2). The data reveals that, on average, in the last 10 minutes of this scenario, we effectively slightly increased video quality (evaluated with VMAF) above 5% while achieving a 25% reduction in bitrate. Particularly noteworthy is the remarkable reduction of over 30% in encoding time, serving as a metric for encoding complexity. This achievement stands out, considering that the anchor implementation is x264 configured for zero latency, which is highly optimized in complexity and aligns with the shift towards more sustainable codecs.

Figure 1: Grafana dashboard displaying KPIs for bitrate reduction, video quality, and encoding complexity during typical DaaS and Thin Client scenarios. This visualization confirms consistent maintenance of video quality (VMAF above 5%) alongside a 25% reduction in bitrate and over 30% reduction in encoding time, using the x264 codec configured for zero latency.

The consistent performance of LCEVC across various common DaaS and Thin Client usage scenarios highlights its versatility. The codec effectively supports a wide range of applications, including customer support, graphic design for gaming, dynamic graph rendering, and color-rich text, effectively showcasing the codec's strong performance across diverse visual requirements. This emphasizes the adaptability and efficiency of the enhancement layer optimizations in preserving quality across a spectrum of content formats.

Figure 2: Screenshots of videos from typical DaaS and Thin Client usage scenarios used to gather KPIs.

To put these improvements into perspective:

• Bitrate reduction: the 25% decrease in bitrate translates to substantial bandwidth savings, potentially allowing for smoother remote desktop experiences even in bandwidth-constrained environments.
• Encoding time improvement: a 33% reduction in encoding time implies lower computational requirements and suggests a potential for reduced latency and improved responsiveness in remote desktop scenarios.

Conclusions

In this blog post, we have detailed the significant progress achieved by integrating MPEG-5 Part 2 LCEVC with FreeRDP. The focus is on highlighting the practical uses and substantial advantages of this integration in remote desktop environments. Our experiments confirmed the effectiveness of this approach in a real-world desktop virtualization setup. The analysis revealed a 25% decrease in bitrate and a 33% reduction in encoding time, demonstrating how LCEVC can optimize operations by significantly reducing bandwidth and computational requirements. This makes it a cost-efficient solution for organizations looking to enhance their IT infrastructure in response to the growing demands of remote work. Furthermore, our approach contributes to environmental sustainability by lowering energy consumption, aligning with regulatory standards, and corporate initiatives for greener technologies.

A key aspect of our approach is the focus on backward compatibility. By building on existing codecs, we ensure our enhancements are compatible with existing communication protocols and hardware. This strategic choice not only facilitates easier adoption without requiring significant infrastructure changes but also extends the life of current hardware investments, making it a cost-effective solution for organizations.

Furthermore, Fluendo’s ongoing work extends beyond enhancement layers and MPEG-5 Part 2 integration. We are actively exploring several key areas to optimize our codecs for DaaS/VDI and Thin Clients:

• Optimization of existing encoders: we are focused on refining existing encoders to deliver superior video quality at lower bitrates, specifically for desktop encoding scenarios. This involves both software tweaks and hardware acceleration to ensure optimal performance.
• Integration of new coding tools: adopting specific Screen Content Coding (SCC) tools is part of our strategy to improve BD-rate efficiency for desktop virtualization, where screen content is a major component.
• Incorporating OS inputs for smarter encoding: by feeding our encoders with feedback from the host server operating system, our encoders can make smarter, faster decisions by eliminating pre-analysis steps at the encoder and thus optimizing the encoding process based on real-time usage data.
• AI-enhanced image quality: by integrating AI at the endpoints, image quality is dynamically enhanced to adjust according to the displayed content. This ensures that even lower-bitrate streams uphold a high-level remote desktop experience.

We invite you to experience these improvements firsthand. Whether you're looking to reduce operational costs, improve user experience, or simply curious about the potential impact on your specific use case, we encourage you to contact us. It's the best way to see how this integration could benefit your organization. Our team is ready to assist you in exploring how this technology can be applied to your unique remote desktop environment.