GUI integration with Qt

Goal

This tutorial teaches how to integrate the Fluendo SDK with the Qt framework. However, the content of this tutorial is relevant to learn how you can integrate it into any other GUI framework. Throughout this explanation, you will learn to:

  • Compile and link a simple Qt application against the Fluendo SDK

  • Render the video frames on a native window of your choice

  • Open, play and pause the video using buttons

Prerequisites

Walkthrough

The code for this tutorial uses C++11 along with Qt5. Qt is a C++-based cross-platform framework which allows us to create GUI applications easily.

Let’s start with our main function. First, we initialize the QApplication setting the AA_Native_Window attribute to ensure that Qt will always use native windows for all widgets. The Fluendo SDK requires a native window where it can draw the decoded video frames. Then, we initialize and set up the GUI, subscribing to the events we’re interested. Finally, we show the main window we create and run the Qt event loop.

main
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
int main(int argc, char *argv[])
{
    QApplication qApplication(argc, argv);

    /* Ensure all windows are native, because the SDK needs them
     * to be native to be able to draw on them */
    qApplication.setAttribute(Qt::AA_NativeWindows);

    /* Initialize the Fluendo SDK */
    flu_initialize();

    /* Initialize the GUI, the flu_player and subscribe to events */
    App app;
    MainWindow window;
    app.window = &window;
    setup_gui(&app);
    subscribe_to_events(&app);
    window.show();

    /* Execute the event loop for the GUI */
    int ret = QApplication::exec();

    /* Shutdown the Fluendo SDK */
    flu_shutdown();

    return ret;
}

Now, our App structure contains the context that we share across our application. It contains the FluPlayer and the 4 Qt widgets we need for this tutorial: the main window, a frame for the video to be rendered in and two buttons to easily select a video and play/pause.

App struct
1
2
3
4
5
6
7
8
struct App
{
    MainWindow *window = nullptr;
    FluPlayer *player = nullptr;
    QFrame *frame = nullptr;
    QPushButton *playButton = nullptr;
    QPushButton *openButton = nullptr;
};

Setting up the GUI is quite straightforward. We create the FluPlayer and add the Open and Play QPushButton’s along with the QFrame to a QGridLayout to ensure the GUI is scaled properly when resizing the window. Apart from that, we add the functionality to the openButton so that when clicked, the user can choose which video to load from a file dialog. After doing so, the play starts automatically. Last, we ensure the player is closed right before closing the main window.

Setup the GUI
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
void setup_gui(App *app)
{
    app->player = flu_player_new();

    app->openButton = new QPushButton("Open");
    app->playButton = new QPushButton("Play");
    app->frame = new QFrame();
    auto layout = new QGridLayout();
    layout->addWidget(app->frame);
    layout->addWidget(app->openButton);
    layout->addWidget(app->playButton);
    app->window->setLayout(layout);
    app->playButton->setEnabled(false);
    app->window->resize(640, 480);

    QObject::connect(app->openButton, &QPushButton::released, [app] {
        auto filename = QFileDialog::getOpenFileUrl(nullptr, "Open video").toString().toStdString();
        std::cout << "Filename selected: " << filename << std::endl;
        flu_player_uri_open(app->player, filename.c_str());
        flu_player_play(app->player);
    });

    /* Close the player before closing the window, because closing the player
     * forces the latest buffers to be pushed and hence it would attempt to
     * draw the last of them into a non-existing window */
    QObject::connect(app->window, &MainWindow::closeEventSignal, [app] {
        flu_player_close(app->player);
        flu_player_unref(app->player);
    });
}

Warning

We need to be very careful when closing the FluPlayer, because doing so pushes the last decoded buffers that will be drawn on our QFrame. For this reason, we need to ensure that when the user closes the main window, we close the player before to stop rendering anything when the frame does not exist anymore. Unfortunately, Qt does not signal an event when the window is requested to be closed, but it does provide a protected method to be overridden called closeEvent. We use this to implement our own MainWindow that emits closeEventSignal when that happens.

Main Window
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
class MainWindow : public QWidget
{
    Q_OBJECT

public:
    explicit MainWindow(QWidget *parent = nullptr) : QWidget(parent) {}

Q_SIGNALS:
    void closeEventSignal();

protected:
    void closeEvent(QCloseEvent *event) override
    {
        Q_EMIT closeEventSignal();
        QWidget::closeEvent(event);
    }
};

/* Ensure the moc parses this .cpp file to generate the QObject code for MainWindow */
#include "qt_integration.moc"

At this point, all that is left is to subscribe to the proper events to finish implementing the logic we want. The event that allows us to render the video texture into a different window than the one the player creates automatically for us is FluPlayerEventRequestTexture. When a video stream is preparing to render the first video frame, it will request a destination drawable to the application. The application should set the target texture pointer into a handle to a native window ID (or window handler) while handling this event. The video rendering component is sometimes able to handle the drawing of black borders and maintain the original video aspect ratio. In most cases, this will work just fine, but it is recommended that the application handles this directly to support special cases, like forcing a specific aspect ratio, video cropping, etc.

We take the native window ID from the QFrame using winId.

Request texture handler
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
    flu_player_event_listener_add(
        app->player, FLU_PLAYER_EVENT_REQUEST_TEXTURE,
        [](FluPlayer *, FluPlayerEvent *event, void *data) -> gboolean {
            auto frame = reinterpret_cast<App *>(data)->frame;
            auto requestTextureEvent = reinterpret_cast<FluPlayerEventRequestTexture *>(event);
            requestTextureEvent->handle = frame->winId();
            requestTextureEvent->handle_aspect_ratio = TRUE;
            return TRUE;
        },
        app);

Also, we subscribe to FLU_PLAYER_EVENT_ERROR so that in case of error, the user gets the message both on the console and through a message box.

Error handler
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
    flu_player_event_listener_add(
        app->player, FLU_PLAYER_EVENT_ERROR,
        [](FluPlayer *, FluPlayerEvent *event, void *data) -> gboolean {
            auto error = reinterpret_cast<FluPlayerEventError *>(event)->error;
            if (error != nullptr && error->message != nullptr)
            {
                QString errorMessage = error->message;
                std::cout << "ERROR: " << errorMessage.toStdString() << std::endl;

                /* We need to post an event so that this is executed in the GUI thread
                 * rather than on this context */
                auto window = reinterpret_cast<App *>(data)->window;
                postWork(window, [errorMessage] {
                    QMessageBox::critical(nullptr, "Error", errorMessage);
                });
            }
            return TRUE;
        },
        app);

Warning

Be careful when interacting with the GUI from any other context different from the main/GUI thread. As a rule of thumb, never change the GUI from any other thread. Check out the Threading basics documentation for more information about it. For this reason, we use the postWork helper function to enqueue the display of the critical message box onto the GUI’s event loop. This will be executed in later iterations of its loop where it’s safe to modify the GUI.

The definition of postWork is quite simple, although it uses a little trick: by connecting to the destroyed signal in a temporary QObject, it enqueues any function that will run into the QObject’s thread.

A similar approach widely used for this is QTimer::singleShot, but it only works with QThread’s. In our case, the handlers for the events are called from threads created by the Fluendo SDK, which are not QThread’s, for obvious reasons.

Post event to object
1
2
3
4
5
6
template <typename T>
void postWork(QObject *obj, T &&func)
{
    QObject tmp;
    QObject::connect(&tmp, &QObject::destroyed, obj, std::forward<T>(func), Qt::QueuedConnection);
}

To give some feedback to the user when the state of the playing changes, we change the text of playButton to either Play or Pause and call the corresponding flu_player_[play/pause]. We use again postWork to change the button, because this is called from a different thread than the GUI one. There is no need to protect neither QObject::connect nor QObject::disconnect because internally Qt already does so.

State handler
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
    flu_player_event_listener_add(
        app->player, FLU_PLAYER_EVENT_STATE,
        [](FluPlayer *, FluPlayerEvent *event, void *data) -> gboolean {
            auto eventState = reinterpret_cast<FluPlayerEventState *>(event);
            auto app = reinterpret_cast<App *>(data);
            if (eventState->state == FLU_PLAYER_STATE_PLAYING)
            {
                app->playButton->disconnect();
                QObject::connect(app->playButton, &QPushButton::released, [app] {
                    flu_player_pause(app->player);
                });
                postWork(app->playButton, [app] {
                    app->playButton->setEnabled(true);
                    app->playButton->setText("Pause");
                });
            }
            else
            {
                app->playButton->disconnect();
                QObject::connect(app->playButton, &QPushButton::released, [app] {
                    flu_player_play(app->player);
                });
                postWork(app->playButton, [app] {
                    app->playButton->setText("Play");
                });
            }
            return TRUE;
        },
        app);

Full source code

Full source code
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
#include <QApplication>
#include <QFileDialog>
#include <QFrame>
#include <QGridLayout>
#include <QMessageBox>
#include <QPushButton>
#include <iostream>

#include "fluendo-sdk.h"

/* Unfortunately, QWidget does not emit a signal when it's asked to
 * be closed, so we need to roll out our own implementation */
class MainWindow : public QWidget
{
    Q_OBJECT

public:
    explicit MainWindow(QWidget *parent = nullptr) : QWidget(parent) {}

Q_SIGNALS:
    void closeEventSignal();

protected:
    void closeEvent(QCloseEvent *event) override
    {
        Q_EMIT closeEventSignal();
        QWidget::closeEvent(event);
    }
};

/* Ensure the moc parses this .cpp file to generate the QObject code for MainWindow */
#include "qt_integration.moc"

struct App
{
    MainWindow *window = nullptr;
    FluPlayer *player = nullptr;
    QFrame *frame = nullptr;
    QPushButton *playButton = nullptr;
    QPushButton *openButton = nullptr;
};

void setup_gui(App *app)
{
    app->player = flu_player_new();

    app->openButton = new QPushButton("Open");
    app->playButton = new QPushButton("Play");
    app->frame = new QFrame();
    auto layout = new QGridLayout();
    layout->addWidget(app->frame);
    layout->addWidget(app->openButton);
    layout->addWidget(app->playButton);
    app->window->setLayout(layout);
    app->playButton->setEnabled(false);
    app->window->resize(640, 480);

    QObject::connect(app->openButton, &QPushButton::released, [app] {
        auto filename = QFileDialog::getOpenFileUrl(nullptr, "Open video").toString().toStdString();
        std::cout << "Filename selected: " << filename << std::endl;
        flu_player_uri_open(app->player, filename.c_str());
        flu_player_play(app->player);
    });

    /* Close the player before closing the window, because closing the player
     * forces the latest buffers to be pushed and hence it would attempt to
     * draw the last of them into a non-existing window */
    QObject::connect(app->window, &MainWindow::closeEventSignal, [app] {
        flu_player_close(app->player);
        flu_player_unref(app->player);
    });
}

/* Helper that uses perfect forwarding to call any function that
 * QObject::connect may accept */
template <typename T>
void postWork(QObject *obj, T &&func)
{
    QObject tmp;
    QObject::connect(&tmp, &QObject::destroyed, obj, std::forward<T>(func), Qt::QueuedConnection);
}

void subscribe_to_events(App *app)
{
    /* FLU_PLAYER_EVENT_REQUEST_TEXTURE handler */
    flu_player_event_listener_add(
        app->player, FLU_PLAYER_EVENT_REQUEST_TEXTURE,
        [](FluPlayer *, FluPlayerEvent *event, void *data) -> gboolean {
            auto frame = reinterpret_cast<App *>(data)->frame;
            auto requestTextureEvent = reinterpret_cast<FluPlayerEventRequestTexture *>(event);
            requestTextureEvent->handle = frame->winId();
            requestTextureEvent->handle_aspect_ratio = TRUE;
            return TRUE;
        },
        app);

    /* FLU_PLAYER_EVENT_ERROR handler */
    flu_player_event_listener_add(
        app->player, FLU_PLAYER_EVENT_ERROR,
        [](FluPlayer *, FluPlayerEvent *event, void *data) -> gboolean {
            auto error = reinterpret_cast<FluPlayerEventError *>(event)->error;
            if (error != nullptr && error->message != nullptr)
            {
                QString errorMessage = error->message;
                std::cout << "ERROR: " << errorMessage.toStdString() << std::endl;

                /* We need to post an event so that this is executed in the GUI thread
                 * rather than on this context */
                auto window = reinterpret_cast<App *>(data)->window;
                postWork(window, [errorMessage] {
                    QMessageBox::critical(nullptr, "Error", errorMessage);
                });
            }
            return TRUE;
        },
        app);

    /* FLU_PLAYER_EVENT_STATE handler */
    flu_player_event_listener_add(
        app->player, FLU_PLAYER_EVENT_STATE,
        [](FluPlayer *, FluPlayerEvent *event, void *data) -> gboolean {
            auto eventState = reinterpret_cast<FluPlayerEventState *>(event);
            auto app = reinterpret_cast<App *>(data);
            if (eventState->state == FLU_PLAYER_STATE_PLAYING)
            {
                app->playButton->disconnect();
                QObject::connect(app->playButton, &QPushButton::released, [app] {
                    flu_player_pause(app->player);
                });
                postWork(app->playButton, [app] {
                    app->playButton->setEnabled(true);
                    app->playButton->setText("Pause");
                });
            }
            else
            {
                app->playButton->disconnect();
                QObject::connect(app->playButton, &QPushButton::released, [app] {
                    flu_player_play(app->player);
                });
                postWork(app->playButton, [app] {
                    app->playButton->setText("Play");
                });
            }
            return TRUE;
        },
        app);
}

int main(int argc, char *argv[])
{
    QApplication qApplication(argc, argv);

    /* Ensure all windows are native, because the SDK needs them
     * to be native to be able to draw on them */
    qApplication.setAttribute(Qt::AA_NativeWindows);

    /* Initialize the Fluendo SDK */
    flu_initialize();

    /* Initialize the GUI, the flu_player and subscribe to events */
    App app;
    MainWindow window;
    app.window = &window;
    setup_gui(&app);
    subscribe_to_events(&app);
    window.show();

    /* Execute the event loop for the GUI */
    int ret = QApplication::exec();

    /* Shutdown the Fluendo SDK */
    flu_shutdown();

    return ret;
}

You can download it here.

Building

This source code along with the rest of tutorials can be compiled using the following commands:

mkdir fluendo-sdk-tutorials && cd fluendo-sdk-tutorials
qmake /opt/fluendo-sdk/share/doc/fluendo-sdk/tutorials/src/qt_integration.pro
make

The qmake project that allow us to link against Fluendo SDK can be downloaded here. The lines that allow us to compile using the Fluendo SDK are:

qmake project
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
unix {
        CONFIG += link_pkgconfig
        PKGCONFIG += fluendo-sdk
        QMAKE_RPATHDIR += $$system(pkg-config fluendo-sdk --variable=libdir)
}

win32 {
        LIBS += -L$$(VS_CERBERO_PREFIX)lib/ -lfluendo-sdk -lglib-2.0
        INCLUDEPATH += $$(VS_CERBERO_PREFIX)include/fluendo-sdk
        INCLUDEPATH += $$(VS_CERBERO_PREFIX)include/glib-2.0
        INCLUDEPATH += $$(VS_CERBERO_PREFIX)lib/glib-2.0/include
}

Conclusions

Wrapping it up, during this tutorial you have learned:

  • How FLU_PLAYER_EVENT_REQUEST_TEXTURE is emitted when the first video frame can be drawn and how in FluPlayerEventRequestTexture we can specify the window ID (or handle) where we want to draw the texture

  • Why we need to close the FluPlayer before the native window is closed to avoid the Fluendo SDK attempting to draw to a non-existing window

  • Why and how we need to ensure that every change to the GUI happens on its thread

  • How to build a Qt application using the Fluendo SDK