How to use FFmpeg with GStreamer?

By Marek Olejnik

You might find yourself in a situation where your application is based on FFmpeg but you can't use it because of codec patents. Or maybe you are using FFmpeg API and just want to improve its performance.

These are precisely the challenges that inspired us to create a new product: Fluendo FFmpeg Enabler. This solution consists of a new library (named libfluffgst) that works as a bridge between FFmpeg and GStreamer. In other words, it allows you to use Fluendo Codec Pack in FFmpeg with all patents needed.

How to use FFmpeg with a GStreamer decoder? Is that even possible?

There's no standard feature. To achieve this, FFmpeg source code has to be modified. As you will see in the example below, it can be done by simply registering new decoders in libavcodec. In general, single decoders can use any external library or even framework. The decoders that we add use GStreamer alongside with Fluendo plugins.

Adding Fluendo decoders to FFmpeg

Libavcodec is an FFmpeg library that is responsible for decoding and encoding. So, how can we make it use a different decoder, i.e. Fluendo AAC decoder, instead of standard ones? The answer is quite simple: registering a new AAC decoder and disabling other AAC decoders. Best way to understand this is with an example.

But before we start, let's define our requirements:

• Add a new AAC decoder
• Decoder is in an external library i.e. libfluffgst

1. Registering the decoder

First thing that has to be done is to register the decoder. It's quite straightforward: all decoders/encoders are registered in allcodecs.c file in libavcodec. The new decoder can be registered like this:

    /* audio codecs */
    REGISTER_DECODER(LIBFLUFFGST_AAC,   libfluffgst_aac);
    REGISTER_ENCDEC (AAC,               aac);
    REGISTER_DECODER(AAC_FIXED,         aac_fixed); 
	

2. Adding configuration options and new dependencies

The new decoder is in an external library, so an additional dependency needs to be added. To make it flexible, it's a good idea to add and configure an option to enable/disable the new decoder. Here is a step-by-step guide on how to modify the configure file:

Add library to list of the external libraries:

    EXTERNAL_LIBRARY_LIST=" 
      libfluffgst 
      avisynth
      ...
	

Add switch to enable/disable the usage of libfluffgst decoders:

    External library support: 
      --enable-libfluffgst     enable Fluendo decoders [no]
      --enable-avisynth        enable reading of AviSynth script files [no]
	

Link to dynamic library when libfluffgst is enabled:

    # these are off by default, so fail if requested and not available: 
      enabled libfluffgst        && add_extralibs -lfluffgst
      enabled avfoundation_indev && { check_header_oc AVFoundation/AVFoundation.h || disable avfoundation_indev; }
            
	

It's the simplest way to add a dependency to libfluffgst. Inside configure script there are few convenient functions defined to check if external libraries are installed in the system (i.e. require_pkg_config)

Lastly, we make the registered decoder depend on external library:

    # decoders / encoders 
    libfluffgst_aac_decoder_deps=libfluffgst
    aac_decoder_select="imdct15 mdct sinewin" 
	

When the decoder is registered, configuration and dependencies are defined, the only thing left is to define the decoder. Above the AAC decoder was registered with name libfluffgst_aac. FFmpeg will expect to have a struct of type AVCodec and name ff_libfluffgst_aac_decoder defined somewhere. It can be defined either on one of the source files of libavcodec or in the external library.

3. Defining the decoder

Definition of AVCodec struct can be found in libavcodec/avcodec.h header. In case of the decoder, the following methods must be specified: init, close, decode and flush. However, we won't dig into details in this post. Code snippet below shows how to define this struct for libfluffgst_aac decoder:

AVCodec ff_libfluffgst_aac_decoder = {
    .name            = "libfluffgst_aac_dec",
    .long_name       = NULL_IF_CONFIG_SMALL("Fluendo AAC decoder"),
    .type            = AVMEDIA_TYPE_AUDIO,
    .id              = AV_CODEC_ID_AAC,
    .priv_data_size  = sizeof(FluFFGstAACDecContext),
    .init            = libfluffgst_aac_dec_init,
    .close           = libfluffgst_aac_dec_close,
    .decode          = libfluffgst_aac_dec_decode,
    .flush           = libfluffgst_aac_dec_flush,
    .sample_fmts     = (const enum AVSampleFormat[]) {
        AV_SAMPLE_FMT_FLTP, AV_SAMPLE_FMT_NONE
    },
    .capabilities    = AV_CODEC_CAP_DELAY,
    .priv_class      = &libfluffgst_aac_dec_class,
};
            
	

Interaction with GStreamer

Now, the question is how is it possible to decode data with Fluendo's GStreamer plugins?
The four methods listed above (init, close, decode, flush) will be called in order to decode the encoded data. This is the bridge between FFmpeg and Gstreamer. FluFFGst library implements them in a way that data passed to these methods is processed by GStreamer pipeline with Fluendo decoder element.

The FluFFGst library provides an implementation that enables using almost all our plugins as a decoders inside FFmpeg.

Patented codecs and licenses

Many of the codecs are based on patented technology. Therefore, users should be aware that a license and payment of applicable royalties is likely required. Our FFmpeg enabler allows organizations that are developing FFmpeg-based applications to easily embed a legal codec solution.

When creating a proprietary application, another problem might appear. Many of the FFmpeg standard codecs are using external free software distributed under GPL license.

Contact us today to learn more about the Fluendo FFmpeg Enabler.

Summary

FluFFGst allows to develop FFmpeg-based codec legal solutions by using FFmpeg libraries licensed under LGPL. The integration is very straightforward. After installing packages it's ready to go. Furthermore, it won't result in any software modifications on already existing systems.

References

• http://en.swpat.org/wiki/Audio-video_patents