Fluster: A framework for multimedia decoder conformance

What is Fluster?

Fluster is a framework built by Fluendo with multimedia decoder conformance in mind. The idea came from the company’s vision of creating a tool capable of running on a developer’s local machine and in a continuous integration (CI) environment. 

The story started in 2020 with Andoni and Pablo. Their main objective was to create a native Python-based tool without dependencies that would be able to fill a gap in the community of multimedia development engineers. We proudly announce that the project’s first official release is out. You can check it out in the project’s GitHub repository.

The tool offers support for both audio and video decoders, audio being the latest addition. The software (SW) design is based on the principle of modularity; as a result, it is fairly easy to add new decoders and test suites.

It is a public, open-source project licensed under LGPLv3 with an active community that regularly contributes new features, improvements, and bug fixes. It is written in Python (compatible with 3.6 and above) and provides a command-line (CLI) application.

There are no required dependencies; however, an optional one enables support for the JUnit XML format of the summary report. As a result, it is OS (Operating System)-independent.

Debian Linux Distribution decided to integrate it into their test tool arsenal for regression detection during development. This is a positive development for the project, showing that its community prizes it. The effort is currently ongoing, and you can take a look here.

Functionality Walkthrough

Let’s take a walk through the integral parts and functionalities of the tool itself. As mentioned previously, Fluster supports audio and video decoders. Here are more details about them:

Codec	Full name	Stream Type	Relevant information
AAC	Advanced Audio Coding	Audio	-
AV1	AOMedia Video 1	Video	https://aomedia.org/av1/
H.264	Advanced Video Coding - AVC	Video	-
H.265	High-Efficiency Video Coding - HEVC	Video	-
H.266	Versatile Video Coding - VVC	Video	-
VP8	-	Video	-
VP9	-	Video	https://www.webmproject.org/vp9

It’s important to mention that various decoders are supported, including GStreamer, FFmpeg-based, and custom reference ones. New ones can be added at will by the user.

Apart from decoders, the other fundamental piece that Fluster needs is test suites. There is at least one reference suite included per decoder. A suite can be associated only with one decoder. Below we provide information about the available suites.

Name	Associated decoder	# Test vectors	Relevant information
ISO_IEC_13818-4_2004	AAC	62	Click here
ISO_IEC_14496-26_2010	AAC	9	Click here
AV1-TEST-VECTORS	AV1	239	Click here
CHROMIUM-8bit-AV1-TEST-VECTORS	AV1	13	Click here
CHROMIUM-10bit-AV1-TEST-VECTORS	AV1	23	Click here
JVT-AVC_V1	H.264	135	Click here
JVT-FR-EXT	H.264	69	Click here
JCT-VC-HEVC_V1	H.265	147	Click here
JVET-VVC_draft6	H.266	282	Click here
VP8-TEST-VECTORS	VP8	61	Click here
VP9-TEST-VECTORS-HIGH	VP9	6	-
VP9-TEST-VECTORS	VP9	305	-

Test suites are written in JSON format and are composed of a collection of reference test vectors. Each vector contains an encoded input file and an expected checksum result, among other information. You can see an example below:

Fluster facilitates the build process of a few reference decoders: AAC, H.264, H.265, and H.266. The process is based on a Makefile and Cmake; therefore, a Linux-based environment is highly recommended, although optional.

The provided CLI application has the following four operation modes.

• List: Print out a list of available test suites and decoders
• Download: When invoked, Fluster downloads and unpacks all test files of existing test suites, leaving them ready for test execution
• Run: Executes conformance testing on all available decoders with all relevant test suites (default mode) or only on instructed decoder with the instructed test suite(s)
• Reference: Fills in the expected results for all test vectors of a specified combination of test suites and decoders.

The table below summarizes decoders, test suites, and vectors available at the time of publication.

Codec	# Decoders	# Test suites	# Test vectors
AAC	2	2	73
AV1	6	3	275
H.264	21	2	204
H.265	24	1	147
H.266	1	1	282
VP8	13	1	61
VP9	12	2	311
Overall	79	12	1353

Below you can see statistical data related to the number of test vectors per codec:

The Road Ahead

Apart from our plans to extend the functionality, we are attentive to what the community would like to see implemented. We welcome contributions that help improve Fluster, so feel free to fork, modify and contribute back to the tool. For more information, check out this guide.