Fluster: A framework for multimedia decoder conformance.

2022年12月29日 07:00

Fluster: A framework for multimedia decoder conformance.

By Michalis Dimopoulos

What is Fluster?

Fluster is a framework built by Fluendo with multimedia decoder conformance in mind. The idea came from the company's vision of creating a tool capable of running on a developer's local machine and in a continuous integration (CI) environment. 

The story started in 2020 with Andoni and Pablo. Their main objective was to create a native python-based tool without dependencies that would be able to fill a gap in the community of multimedia development engineers. We proudly announce that the project's first official release is out. You can check it out in the project's GitHub repository.

The tool offers support for both audio and video decoders, audio being the latest addition. The software (SW) design is based on the principle of modularity; as a result, it is fairly easy to add new decoders and test suites.

It is a public, open-source project licensed under LGPLv3 with an active community that regularly contributes new features, improvements, and bug fixes. It is written in Python (compatible with 3.6 and above) and provides a command-line (CLI) application.

There are no required dependencies; however, an optional one enables support for the JUnit XML format of the summary report. As a result, it is OS (Operating System)-independent.

Debian Linux Distribution decided to integrate it into their test tool arsenal for regression detection during development. This is a positive development for the project, showing that its community prizes it. The effort is currently ongoing, and you can take a look here.

Functionality Walkthrough

Let's take a walk through the integral parts and functionalities of the tool itself. As mentioned previously, Fluster supports audio and video decoders. Here are more details about them:

Codec Full name Stream Type Relevant information
AAC Advanced Audio Coding Audio -
AV1 AOMedia Video 1 Video https://aomedia.org/av1/ 
H.264 Advanced Video Coding - AVC Video -
H.265 High-Efficiency Video Coding - HEVC Video -
H.266 Versatile Video Coding - VVC Video -
VP8 - Video -
VP9 - Video https://www.webmproject.org/vp9 

 

 

 

 

 

 

It's important to mention that various decoders are supported, including GStreamer, FFmpeg-based, and custom reference ones. New ones can be added at will by the user.

Apart from decoders, the other fundamental piece that Fluster needs is test suites. There is at least one reference suite included per decoder. A suite can be associated only with one decoder. Below we provide information about the available suites.
 

Name Associated decoder # Test vectors Relevant information
ISO_IEC_13818-4_2004 AAC 62 Click here
ISO_IEC_14496-26_2010 AAC 9 Click here 
AV1-TEST-VECTORS AV1 239 Click here 
CHROMIUM-8bit-AV1-TEST-VECTORS AV1 13 Click here
CHROMIUM-10bit-AV1-TEST-VECTORS AV1 23 Click here
JVT-AVC_V1 H.264 135 Click here
JVT-FR-EXT H.264 69 Click here
JCT-VC-HEVC_V1 H.265 147 Click here
JVET-VVC_draft6 H.266 282 Click here
VP8-TEST-VECTORS VP8 61 Click here
VP9-TEST-VECTORS-HIGH VP9 6 -
VP9-TEST-VECTORS VP9 305 -

 

Test suites are written in JSON format and are composed of a collection of reference test vectors. Each vector contains an encoded input file and an expected checksum result, among other information. You can see an example below:

Fluster facilitates the build process of a few reference decoders: AAC, H.264, H.265, and H.266. The process is based on a Makefile and Cmake; therefore, a Linux-based environment is highly recommended, although optional.

The provided CLI application has the following four operation modes.

  • List: Print out a list of available test suites and decoders

  • Download: When invoked, Fluster downloads and unpacks all test files of existing test suites, leaving them ready for test execution

  • Run: Executes conformance testing on all available decoders with all relevant test suites (default mode) or only on instructed decoder with the instructed test suite(s)

  • Reference: Fills in the expected results for all test vectors of a specified combination of test suites and decoders.

The table below summarizes decoders, test suites, and vectors available at the time of publication.
 

Codec # Decoders # Test suites # Test vectors
AAC 2 2 73
AV1 6 3 275
H.264 21 2 204
H.265 24 1 147
H.266 1 1 282
VP8 13 1 61
VP9 12 2 311
Overall 79 12 1353


Below you can see statistical data related to the number of test vectors per codec:

The Road Ahead

Apart from our plans to extend the functionality, we are attentive to what the community would like to see implemented. We welcome contributions that help improve Fluster, so feel free to fork, modify and contribute back to the tool. For more information, check out this guide.