Stream Management

Goal

A media file may contain different types of streams. This tutorial explains how to enable and disable them. By the end of the tutorial you will have learned:

  • What is a container and a stream

  • Collect the information of the streams within a container

  • Enable or disable them while playing

Introduction

As briefly mentioned in previous tutorials, different multimedia containers can hold different sorts of information within. While some of them as the WAV format can only hold audio information, some others as MP4 can store both video and audio. Then, there are others such as Matroska that are independent of the coding format and can hold an unlimited number of video, audio, picture and subtitle tracks (or streams) in one single file. This is quite convenient, since with the same container, a video player may change the quality of the video (or even show 2 video streams at the same time), the language of the audio and even the subtitles.

Walkthrough

We kick it off by refactoring the previous code so that we can init and clean the code with the _initialize and _cleanup functions:

Initialize and clean up
static void _cleanup(App *app)
{
    /* Remove the keyboard handler, close the player, unref it, unref the main loop
       and shutdown the Fluendo SDK */
    g_source_remove(app->keyboard_source_id);
    flu_player_close(app->player);
    flu_player_unref(app->player);
    g_main_loop_unref(app->main_loop);
    flu_shutdown();
}

static void _initialize(App *app, int argc, const char **argv)
{
    /* Initialize the Fluendo SDK and create the player */
    flu_initialize();
    app->player = flu_player_new();

    /* Set the URI to the first argument if given */
    app->uri = "http://ftp.halifax.rwth-aachen.de/blender/demo/movies/ToS/tears_of_steel_720p.mov";
    if (argc > 1)
    {
        app->uri = argv[1];
        g_print("Setting video stream URI: %s\n", app->uri);
    }
}

Our main function initializes the Fluendo SDK as usual and starts playing the video. We are also going to subscribe to some events exactly as we did in the Event Management tutorial. Then, to be able to read the commands that change the streams status, we use what we learned in the Playback Controls tutorial to set up a keyboard handle.

Keyboard handler
static gboolean _handle_keyboard(GIOChannel *source, GIOCondition cond, gpointer data)
{
    gchar *str = NULL;
    gsize terminator_pos = 0;
    App *app = (App *)data;

    if (g_io_channel_read_line(source, &str, NULL, &terminator_pos, NULL) == G_IO_STATUS_NORMAL)
    {
        str[terminator_pos] = 0;

        switch (str[0])
        {
        case 'q':
            g_print("Exiting...\n");
            g_main_loop_quit(app->main_loop);
            break;

        case 'v':
            g_print("Toggling video streams\n");
            _toggle_streams(app, "video", flu_player_video_streams_get, flu_player_video_active_streams_set);
            break;

        case 'a':
            g_print("Toggling audio streams\n");
            _toggle_streams(app, "audio", flu_player_audio_streams_get, flu_player_audio_active_streams_set);
            break;

        case 't':
            g_print("Toggling text streams\n");
            _toggle_streams(app, "text", flu_player_text_streams_get, flu_player_text_active_streams_set);
            break;

        case 'i':
            _display_streams_info(app->player);
            break;

        default:
            g_print("Unrecognized command '%s':\n"
                    "   q - quit the player\n"
                    "   v - toggle all video streams\n"
                    "   a - toggle all audio streams\n"
                    "   t - toggle all text streams\n"
                    "   i - display streams info\n",
                    str);
            break;
        }
    }

    g_free(str);

    return TRUE;
}

Now, let’s start with the _toggle_streams call for every kind of stream. This function is responsible of toggling all the streams of the same kind. This means that if a container has 2 video streams, both videos are disabled or enabled when using the v shortkey. The function accepts as the last two parameters two functions: one to get a list of streams and another one to set the active streams. Then, we use flu_stream_is_active to distinguish the current state of one stream to know whether we need to activate or deactivate all of the same kind.

Note

Take into account that we are using only the state of one stream of some sort to change all of them. This is ok for this example since we are always toggling the streams of the same type altogether.

Stream getter and setter prototypes
typedef GList *(*StreamsGet)(FluPlayer *player);
typedef void (*ActiveStreamsSet)(FluPlayer *player, GList *streams);
Toggle streams
static void _toggle_streams(App *app, const char *name, StreamsGet streams_get, ActiveStreamsSet active_streams_set)
{
    GList *streams = streams_get(app->player);
    if (streams && streams->data && !flu_stream_is_pending(streams->data))
    {
        if (!flu_stream_is_active(streams->data))
        {
            g_print("%s stream %p is not active. Activating it...\n", name, streams->data);
            active_streams_set(app->player, streams);
        }
        else
        {
            g_print("Disabling all %s streams\n", name);
            active_streams_set(app->player, NULL);
            flu_stream_list_free(streams);
        }
    }
    else
    {
        g_print("No %s streams available to toggle\n", name);
    }
}

Attention

Beware the flu_player_<type>_active_streams_set transfer information. It is transfer full, which means that the function will take ownership of the GList passed and it will deallocate it itself when necessary. This is the reason we only call flu_stream_list_free when we don’t actually pass (and transfer) any GList.

Now’s the time of the _display_streams_info function. The purpose of this function is to show a brief description of each of the available streams. To do so, it uses the flu_player_<type>_streams_get API calls to collect the data for video, audio, text and data streams. The calls are, respectively: flu_player_video_streams_get, flu_player_audio_streams_get, flu_player_text_streams_get and flu_player_data_streams_get. All of them return a GList of FluStream that we need to free after using them with flu_stream_list_free. We define a _print_stream_info_generic that accepts the list of streams along with the name of the stream and a function to show information for each of them. We define the prototype of this function as follows:

Print stream info prototype
typedef void (*PrintStreamInfo)(const FluStreamInfo *stream_info, guint idx);

Through flu_stream_info_get, we can collect the FluStreamInfo for each stream. We pass this structure to each of our print functions, one per stream type.

Display streams info
static void _print_stream_info_generic(GList *streams, const char *name, PrintStreamInfo print_stream_info)
{
    guint size = g_list_length(streams);
    if (size > 0)
    {
        guint idx = 0;
        for (GList *item = streams; item; item = item->next)
        {
            FluStream *stream = (FluStream *)item->data;
            const FluStreamInfo *stream_info = NULL;
            if (!flu_stream_info_get(stream, &stream_info))
            {
                g_print("   %s stream #%03u => Error: cannot fetch stream information\n",
                        name, idx++);
            }
            else
            {
                print_stream_info(stream_info, idx++);
            }
        }
    }
    else
    {
        g_print("   No %s streams available.\n", name);
    }
}

static void _display_streams_info(FluPlayer *player)
{
    if (player)
    {
        GList *streams = flu_player_video_streams_get(player);
        _print_stream_info_generic(streams, "video", _print_stream_video_info);
        flu_stream_list_free(streams);

        streams = flu_player_audio_streams_get(player);
        _print_stream_info_generic(streams, "audio", _print_stream_audio_info);
        flu_stream_list_free(streams);

        streams = flu_player_text_streams_get(player);
        _print_stream_info_generic(streams, "text", _print_stream_text_info);
        flu_stream_list_free(streams);

        streams = flu_player_data_streams_get(player);
        _print_stream_info_generic(streams, "data", _print_stream_data_info);
        flu_stream_list_free(streams);
    }
}

Then, we define the different print methods that show the elements we are more interested in from the FluStreamInfo structure. Within this structure, we need to choose the proper type of the FluStreamInfoData depending on the kind of stream.

Print for each stream
#define STRING_OR_NONE(str) ((str) && (str)[0]) ? (str) : "none"

static void _print_stream_video_info(const FluStreamInfo *stream_info, guint idx)
{
    g_print("   video stream #%03u => %s %dx%d, fps: %d, bps: %d\n",
            idx,
            STRING_OR_NONE(stream_info->data.video.codec),
            stream_info->data.video.width,
            stream_info->data.video.height,
            stream_info->data.video.fps_n,
            stream_info->data.video.bitrate);
}

static void _print_stream_audio_info(const FluStreamInfo *stream_info, guint idx)
{
    g_print("   audio stream #%03u => %s, rate: %d Hz, channels: %d\n",
            idx,
            STRING_OR_NONE(stream_info->data.audio.codec),
            stream_info->data.audio.rate,
            stream_info->data.audio.channels);
}

static void _print_stream_text_info(const FluStreamInfo *stream_info, guint idx)
{
    g_print("   text stream #%03u => %s lang: %s, ISO: %s",
            idx,
            STRING_OR_NONE(stream_info->data.text.codec),
            STRING_OR_NONE(stream_info->data.text.language),
            STRING_OR_NONE(stream_info->data.text.iso));
}

static void _print_stream_data_info(const FluStreamInfo *stream_info, guint idx)
{
    g_print("   data stream #%03u => data-type: %d , mime-type: %s",
            idx,
            stream_info->data.data.type,
            STRING_OR_NONE(stream_info->data.data.mime));
}
Output
Info: video stream 0x7f20d0007030 has become active.
Info: audio stream 0x7f20d00071b0 has become active.
v
Toggling video streams
Disabling all video streams
Info: video stream 0x7f20d0007030 has become inactive.
v
Toggling video streams
video stream 0x7f20d0007030 is not active. Activating it…
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
Info: video stream 0x7f20d0007030 has become active.
a
Toggling audio streams
Disabling all audio streams
Info: audio stream 0x7f20d00071b0 has become inactive.
a
Toggling audio streams
audio stream 0x7f20d00071b0 is not active. Activating it…
Info: audio stream 0x7f20d00071b0 has become active.
i
video stream #000 => H.264/AVC video 1280x534, fps: 24, bps: 0
audio stream #000 => MPEG 1 Audio, Layer 3 (MP3), rate: 44100 Hz, channels: 2
No text streams available.
No data streams available.
q

Full source code

Full source code
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
#include <fluendo-sdk.h>
#include <stdio.h>

typedef struct
{
    GMainLoop *main_loop;
    FluPlayer *player;
    const gchar *uri;
    guint keyboard_source_id;
} App;

typedef void (*PrintStreamInfo)(const FluStreamInfo *stream_info, guint idx);
typedef GList *(*StreamsGet)(FluPlayer *player);
typedef void (*ActiveStreamsSet)(FluPlayer *player, GList *streams);

#define STRING_OR_NONE(str) ((str) && (str)[0]) ? (str) : "none"

static void _print_stream_video_info(const FluStreamInfo *stream_info, guint idx)
{
    g_print("   video stream #%03u => %s %dx%d, fps: %d, bps: %d\n",
            idx,
            STRING_OR_NONE(stream_info->data.video.codec),
            stream_info->data.video.width,
            stream_info->data.video.height,
            stream_info->data.video.fps_n,
            stream_info->data.video.bitrate);
}

static void _print_stream_audio_info(const FluStreamInfo *stream_info, guint idx)
{
    g_print("   audio stream #%03u => %s, rate: %d Hz, channels: %d\n",
            idx,
            STRING_OR_NONE(stream_info->data.audio.codec),
            stream_info->data.audio.rate,
            stream_info->data.audio.channels);
}

static void _print_stream_text_info(const FluStreamInfo *stream_info, guint idx)
{
    g_print("   text stream #%03u => %s lang: %s, ISO: %s",
            idx,
            STRING_OR_NONE(stream_info->data.text.codec),
            STRING_OR_NONE(stream_info->data.text.language),
            STRING_OR_NONE(stream_info->data.text.iso));
}

static void _print_stream_data_info(const FluStreamInfo *stream_info, guint idx)
{
    g_print("   data stream #%03u => data-type: %d , mime-type: %s",
            idx,
            stream_info->data.data.type,
            STRING_OR_NONE(stream_info->data.data.mime));
}

static void _print_stream_info_generic(GList *streams, const char *name, PrintStreamInfo print_stream_info)
{
    guint size = g_list_length(streams);
    if (size > 0)
    {
        guint idx = 0;
        for (GList *item = streams; item; item = item->next)
        {
            FluStream *stream = (FluStream *)item->data;
            const FluStreamInfo *stream_info = NULL;
            if (!flu_stream_info_get(stream, &stream_info))
            {
                g_print("   %s stream #%03u => Error: cannot fetch stream information\n",
                        name, idx++);
            }
            else
            {
                print_stream_info(stream_info, idx++);
            }
        }
    }
    else
    {
        g_print("   No %s streams available.\n", name);
    }
}

static void _display_streams_info(FluPlayer *player)
{
    if (player)
    {
        GList *streams = flu_player_video_streams_get(player);
        _print_stream_info_generic(streams, "video", _print_stream_video_info);
        flu_stream_list_free(streams);

        streams = flu_player_audio_streams_get(player);
        _print_stream_info_generic(streams, "audio", _print_stream_audio_info);
        flu_stream_list_free(streams);

        streams = flu_player_text_streams_get(player);
        _print_stream_info_generic(streams, "text", _print_stream_text_info);
        flu_stream_list_free(streams);

        streams = flu_player_data_streams_get(player);
        _print_stream_info_generic(streams, "data", _print_stream_data_info);
        flu_stream_list_free(streams);
    }
}

static void _toggle_streams(App *app, const char *name, StreamsGet streams_get, ActiveStreamsSet active_streams_set)
{
    GList *streams = streams_get(app->player);
    if (streams && streams->data && !flu_stream_is_pending(streams->data))
    {
        if (!flu_stream_is_active(streams->data))
        {
            g_print("%s stream %p is not active. Activating it...\n", name, streams->data);
            active_streams_set(app->player, streams);
        }
        else
        {
            g_print("Disabling all %s streams\n", name);
            active_streams_set(app->player, NULL);
            flu_stream_list_free(streams);
        }
    }
    else
    {
        g_print("No %s streams available to toggle\n", name);
    }
}

static gboolean _handle_keyboard(GIOChannel *source, GIOCondition cond, gpointer data)
{
    gchar *str = NULL;
    gsize terminator_pos = 0;
    App *app = (App *)data;

    if (g_io_channel_read_line(source, &str, NULL, &terminator_pos, NULL) == G_IO_STATUS_NORMAL)
    {
        str[terminator_pos] = 0;

        switch (str[0])
        {
        case 'q':
            g_print("Exiting...\n");
            g_main_loop_quit(app->main_loop);
            break;

        case 'v':
            g_print("Toggling video streams\n");
            _toggle_streams(app, "video", flu_player_video_streams_get, flu_player_video_active_streams_set);
            break;

        case 'a':
            g_print("Toggling audio streams\n");
            _toggle_streams(app, "audio", flu_player_audio_streams_get, flu_player_audio_active_streams_set);
            break;

        case 't':
            g_print("Toggling text streams\n");
            _toggle_streams(app, "text", flu_player_text_streams_get, flu_player_text_active_streams_set);
            break;

        case 'i':
            _display_streams_info(app->player);
            break;

        default:
            g_print("Unrecognized command '%s':\n"
                    "   q - quit the player\n"
                    "   v - toggle all video streams\n"
                    "   a - toggle all audio streams\n"
                    "   t - toggle all text streams\n"
                    "   i - display streams info\n",
                    str);
            break;
        }
    }

    g_free(str);

    return TRUE;
}

static gboolean _player_on_error(FluPlayer *player, FluPlayerEvent *event, gpointer data)
{
    FluPlayerEventError *ev = (FluPlayerEventError *)event;
    App *app = (App *)data;

    g_print("Error: %s\nDetailed description: %s.\n", ev->error->message, ev->dbg);
    g_main_loop_quit(app->main_loop);

    return TRUE;
}

static const gchar *_stream_type_name(FluStream *stream)
{
    switch (flu_stream_type_get(stream))
    {
    case FLU_STREAM_TYPE_VIDEO:
        return "video";
    case FLU_STREAM_TYPE_AUDIO:
        return "audio";
    case FLU_STREAM_TYPE_TEXT:
        return "text";
    case FLU_STREAM_TYPE_DATA:
        return "data";
    default:
        return "unknown";
    }
}

static gboolean _player_on_stream_state_changed(FluPlayer *player, FluPlayerEvent *event, gpointer data)
{
    FluPlayerEventStreamStateChanged *ev = (FluPlayerEventStreamStateChanged *)event;

    if (ev->stream && !flu_stream_is_pending(ev->stream))
    {
        gboolean active = flu_stream_is_active(ev->stream);
        g_print("Info: %s stream %p has become %s.\n",
                _stream_type_name(ev->stream),
                ev->stream,
                active ? "active" : "inactive");
    }

    return TRUE;
}

static void _cleanup(App *app)
{
    /* Remove the keyboard handler, close the player, unref it, unref the main loop
       and shutdown the Fluendo SDK */
    g_source_remove(app->keyboard_source_id);
    flu_player_close(app->player);
    flu_player_unref(app->player);
    g_main_loop_unref(app->main_loop);
    flu_shutdown();
}

static void _initialize(App *app, int argc, const char **argv)
{
    /* Initialize the Fluendo SDK and create the player */
    flu_initialize();
    app->player = flu_player_new();

    /* Set the URI to the first argument if given */
    app->uri = "http://ftp.halifax.rwth-aachen.de/blender/demo/movies/ToS/tears_of_steel_720p.mov";
    if (argc > 1)
    {
        app->uri = argv[1];
        g_print("Setting video stream URI: %s\n", app->uri);
    }
}

int main(int argc, const char **argv)
{
    App app = {0};

    _initialize(&app, argc, argv);

    /* Ask the player to open the video and start playing it */
    flu_player_uri_open(app.player, app.uri);
    flu_player_play(app.player);

    /* Create the main loop that will handle the events we subscribe to */
    app.main_loop = g_main_loop_new(NULL, FALSE);

    /* Subscribe to the desired events, passing the App struct to the ones that need it */
    flu_player_event_listener_add(app.player, FLU_PLAYER_EVENT_ERROR, _player_on_error, &app);
    flu_player_event_listener_add(app.player, FLU_PLAYER_EVENT_STREAM_STATE_CHANGED, _player_on_stream_state_changed, &app);

    /* Set a keyboard handler to capture strokes in our main thread */
    GIOChannel *io_stdin = g_io_channel_unix_new(fileno(stdin));
    app.keyboard_source_id = g_io_add_watch(io_stdin, G_IO_IN, (GIOFunc)_handle_keyboard, &app);
    g_io_channel_unref(io_stdin);

    /* Start running the main_loop. This will run the loop until
       g_main_loop_quit is called on it. */
    g_main_loop_run(app.main_loop);

    _cleanup(&app);

    return 0;
}

You can download it here.

Building

This source code along with the rest of tutorials can be compiled using the following commands.

On Linux:

mkdir fluendo-sdk-tutorials && cd fluendo-sdk-tutorials
meson /opt/fluendo-sdk/share/doc/fluendo-sdk/tutorials/src
ninja

On Windows:

mkdir fluendo-sdk-tutorials
cd fluendo-sdk-tutorials
meson C:\fluendo-sdk\<version>\<x86/x86_64>\share\doc\fluendo-sdk\tutorials\src
ninja

To generate a Visual Studio project, you can pass the --backend=vs option to meson.

Conclusions

Wrapping it up, in this tutorial you have learned: