Fluster: A framework for multimedia decoder conformance

What is Fluster?

Fluster is a framework built by Fluendo with multimedia decoder conformance in mind. The idea came from the company’s vision of creating a tool capable of running on a developer’s local machine and in a continuous integration (CI) environment. 

The story started in 2020 with Andoni and Pablo. Their main objective was to create a native Python-based tool without dependencies that would be able to fill a gap in the community of multimedia development engineers. We proudly announce that the project’s first official release is out. You can check it out in the project’s GitHub repository.

The tool offers support for both audio and video decoders, audio being the latest addition. The software (SW) design is based on the principle of modularity; as a result, it is fairly easy to add new decoders and test suites.

It is a public, open-source project licensed under LGPLv3 with an active community that regularly contributes new features, improvements, and bug fixes. It is written in Python (compatible with 3.6 and above) and provides a command-line (CLI) application.

There are no required dependencies; however, an optional one enables support for the JUnit XML format of the summary report. As a result, it is OS (Operating System)-independent.

Debian Linux Distribution decided to integrate it into their test tool arsenal for regression detection during development. This is a positive development for the project, showing that its community prizes it. The effort is currently ongoing, and you can take a look here.

Functionality Walkthrough

Let’s take a walk through the integral parts and functionalities of the tool itself. As mentioned previously, Fluster supports audio and video decoders. Here are more details about them:

It’s important to mention that various decoders are supported, including GStreamer, FFmpeg-based, and custom reference ones. New ones can be added at will by the user.

Apart from decoders, the other fundamental piece that Fluster needs is test suites. There is at least one reference suite included per decoder. A suite can be associated only with one decoder. Below we provide information about the available suites.

Test suites are written in JSON format and are composed of a collection of reference test vectors. Each vector contains an encoded input file and an expected checksum result, among other information. You can see an example below:

Fluster facilitates the build process of a few reference decoders: AAC, H.264, H.265, and H.266. The process is based on a Makefile and Cmake; therefore, a Linux-based environment is highly recommended, although optional.

The provided CLI application has the following four operation modes.

• List: Print out a list of available test suites and decoders
• Download: When invoked, Fluster downloads and unpacks all test files of existing test suites, leaving them ready for test execution
• Run: Executes conformance testing on all available decoders with all relevant test suites (default mode) or only on instructed decoder with the instructed test suite(s)
• Reference: Fills in the expected results for all test vectors of a specified combination of test suites and decoders.

The table below summarizes decoders, test suites, and vectors available at the time of publication.

Below you can see statistical data related to the number of test vectors per codec:

The Road Ahead

Apart from our plans to extend the functionality, we are attentive to what the community would like to see implemented. We welcome contributions that help improve Fluster, so feel free to fork, modify and contribute back to the tool. For more information, check out this guide.