Open Menu
Face anonymization with real-time re-identification

Face anonymization with real-time re-identification

User Name

Written by

Pawarit Jamjod

July 8, 2026

At Fluendo, we specialize in multimedia processing and have extensive experience working with GStreamer and other industry-leading technologies. One of the challenges we’ve tackled is anonymizing faces in video content — a critical need for industries such as broadcasting, security and surveillance, smart cities, and any organization that handles footage of real people.

We have developed an AI-powered solution that automatically detects, recognizes, and selectively anonymizes faces in video streams. Rather than blurring everyone indiscriminately, our system can re-identify the people you choose to keep visible while protecting everyone else — and it runs entirely on the edge, ensuring privacy, cost efficiency, and full control over sensitive biometric data.

Table of contents

The challenge of face anonymization

Anonymizing faces in video sounds simple — just blur them — but doing it well presents several real challenges.

The first is selectivity. Blurring every face in a frame is easy, but most real use cases need nuance: a broadcaster may want to anonymize the crowd while keeping presenters visible, or a security team may need to hide unknown individuals while still recognizing authorized staff. That requires not just detecting faces, but recognizing who they belong to.

The second challenge is consistency over time. A person walks behind a pillar, turns their head, or is briefly hidden by another face, and a naive system loses track of them, causing the anonymization to flicker on and off. Maintaining a stable identity for each person across an entire video is far harder than processing a single still image.

The third challenge is the nature of faces “in the wild.” Real footage contains extreme angles, low-resolution and grainy surveillance imagery, changing lighting, and partial occlusions like masks. Recognition accuracy can fall sharply as a face turns away from the camera — an effect we call the “pose cliff” — and a robust system has to work despite it.

Finally, there is privacy itself. Face embeddings and facial crops are classified as sensitive biometric data under GDPR Article 9, CCPA, and similar regulations. Many solutions send this data to the cloud for processing, which introduces both privacy exposure and ongoing costs. For many organizations, keeping all processing offline is not a preference but a requirement.

Our solution is designed around exactly these constraints: selective, consistent, robust to difficult footage, and fully offline.

Our AI-powered face anonymization system

We designed our face anonymization solution to be simple, robust, scalable, and adaptable to diverse business needs. By combining state-of-the-art face detection, alignment, and re-identification with intelligent tracking, our system accurately identifies and anonymizes people across a wide range of video content. Built as a native GStreamer plugin with a clean user interface, it integrates seamlessly into existing multimedia workflows, whether deployed on-premises or at the edge — and it runs comfortably on modest hardware, achieving real-time performance of 15–30 frames per second on an Intel i5 CPU paired with an Nvidia GeForce MX550 GPU.

For the client, this translates into a powerful tool that protects privacy, streamlines compliance, and gives full control over how identities are handled. Whether anonymizing crowds in broadcast footage, protecting bystanders in security recordings, or selectively revealing authorized personnel, our solution lets clients extract value from their video while keeping sensitive biometric data entirely under their control.

Key features of our solution

  • On-the-edge processing: No internet connection is required, ensuring privacy, eliminating cloud costs, and supporting GDPR-friendly deployments.
  • Selective anonymization: Choose exactly which faces to anonymize — unknown people, registered people, everyone, or no one — and switch between a pixelate or blur effect at any intensity.
  • Real-time re-identification: Our system recognizes registered identities on the fly and keeps them consistently labeled across frames.
  • On-demand and automatic registration: Add a new identity from a single image, or let the system automatically learn and register the faces it sees.
  • Robust tracking: Multiple tracker variants keep identities stable through occlusions, rapid motion, and challenging camera conditions.
  • GStreamer-native plugin: The system runs as a standard GStreamer element with a Qt interface, making it easy to drop into existing multimedia pipelines.

The overall design and structure of the solution are illustrated in Figure 1, which provides a visual representation of the key components, their interactions, and how they collectively support the system’s functionality. This diagram helps to clarify the architecture by outlining the relationships between different stages, data flows, and integration points within the solution.

Solution Diagram Figure 1: System workflow — from video input to the final anonymized output

How does it work?

Below, we explain how our system detects, recognizes, tracks, and anonymizes faces in video content, as shown in Figure 1.

1. Input processing & the GStreamer pipeline

Our system is built as a native GStreamer plugin, so it slots directly into standard multimedia pipelines. Video frames flow into the plugin, which processes each frame in real time and passes an anonymized frame back out. Because it is a GStreamer element, it works with the wide range of input sources and formats that GStreamer already supports — files, live streams, and capture devices alike.

To keep performance high, the system can run full detection and recognition every frame, or skip frames and reuse the most recent results on the frames in between, trading a small amount of responsiveness for higher throughput when needed.

2. Face detection – finding every face in the frame

The first step in every frame is finding the faces. We use a fast and accurate face detector, to locate every face and return its bounding box along with a set of facial keypoints (eyes, nose, and mouth corners). This detector was selected after benchmarking several state-of-the-art alternatives for the best balance of accuracy and speed.

The keypoints returned are not just for locating the face — they are the foundation for the next, crucial step.

3. Face alignment – solving the “pose cliff”

Recognition works best on a clean, front-facing view of a face. In real video, though, people rarely look straight at the camera. As a face turns to the side, recognition accuracy can drop sharply — the “pose cliff.”

To counter this, we align each detected face before recognizing it, using the keypoints from the detection step to rotate and normalize the face into a consistent, upright orientation. We benchmarked several approaches for this, from lightweight five-point geometric alignment to dense 68-point 3D facial modelling that can reconstruct and re-render a frontal view of an angled face. The result is a normalized face image that gives the recognition step the best possible chance of an accurate match.

4. Feature extraction – generating a unique face signature

Once a face is aligned, we convert it into a compact numerical signature called an embedding. Using a modern transformer-based recognition model, each face is reduced to a 512-dimensional vector that captures the distinctive characteristics of that individual — independent of lighting, minor angle changes, or image quality.

This embedding is the face’s “fingerprint.” Two images of the same person produce embeddings that point in nearly the same direction, while different people produce embeddings that point elsewhere. That property is what makes reliable recognition possible.

To recognize a face, we compare its embedding against a gallery of registered identities. We use cosine similarity — a measure of how closely two embeddings point in the same direction — to score each comparison, and an efficient assignment step to match the faces in the current frame to the identities in the gallery.

The matching process follows a structured approach:

  • Confident match: If a face’s embedding is similar enough to a registered identity, we recognize it and label it with that identity.
  • Confidence scoring: Every comparison produces a similarity score, and only matches above a configurable threshold are accepted, keeping false identifications low.
  • Unknown handling: If no registered identity is similar enough, the face is treated as unknown — which, depending on your settings, is exactly the category you may want to anonymize.

By structuring matching this way, the system recognizes the people it should while confidently rejecting those it shouldn’t.

6. Tracking – keeping identities consistent over time

Recognizing a face in a single frame is only half the problem. To anonymize people smoothly across an entire video, the system has to keep each person’s identity stable from one frame to the next — even when they turn away, are briefly hidden, or cross paths with someone else.

This is the job of our tracking layer. Each person becomes a “track” that carries their identity forward through time. A track state machine manages each track’s lifecycle — confirming a track once it has been seen consistently, keeping it alive briefly when a face temporarily disappears, and removing it only when the person is truly gone. This prevents the anonymization from flickering and ensures that when someone reappears after a brief occlusion, they are recognized as the same person rather than a new one.

We developed and benchmarked several tracker variants — an appearance-based tracker that relies purely on the strength of the face embeddings, and motion-based variants that also predict where each face will move next. Each suits different conditions, from chaotic, fast-cut footage to steady fixed-camera environments.

7. Identity registration – teaching the system who to recognize

The system can only recognize people it has been introduced to, and registering them is simple. To add someone to the gallery, you provide an image; the system detects the face, aligns it, extracts its embedding, and stores that identity for future recognition — all without interrupting the live video.

For convenience, the system also supports automatic registration. As it tracks a confirmed face over several frames, it averages those views into a single, stable signature before saving it. This smooths out the noise of any single frame — a moment of motion blur or an awkward angle — and produces a far more reliable identity template than a single snapshot would.

Registration flow Figure 2: The identity registration flow - how to recognize an individual

8. Applying the privacy effect

With every face in the frame detected, recognized, and tracked, the final step is anonymization. Here you are fully in control. The system can anonymize unknown faces only (privacy mode), registered faces only, everyone, or no one — and you can choose between a pixelate or a blur effect and dial its intensity up or down.

Because anonymization is tied to each person’s tracked identity, the effect follows them smoothly through the video. An unknown person stays anonymized as they move across the scene, while a registered, authorized individual remains clearly visible — exactly the selective control that blanket blurring can’t provide.

Figure 3: Enhanced reid anonymizer demo

Performance & evaluation

Our face anonymization system has been thoroughly benchmarked against state-of-the-art alternatives, demonstrating excellent tracking quality and detection accuracy under demanding conditions. We evaluated it on a challenging benchmark of diverse video clips featuring dense crowds, frequent shot changes, extreme variations in face scale, severe occlusions, and broad demographic diversity — the kind of difficult footage that real-world deployments must handle.

Tracking accuracy

Tracking quality is measured primarily with HOTA (Higher Order Tracking Accuracy), a unified metric that balances how well a system detects faces with how consistently it maintains each person’s identity over time.

Our in-house appearance-based pipeline achieved a HOTA of 52.41%, outperforming a leading state-of-the-art tracker by more than 13 percentage points. The advantage comes almost entirely from superior identity consistency: by relying on the strength of high-quality face embeddings rather than rigid spatial rules, our system keeps people’s identities coherent over far longer stretches of video — exactly what matters most for clean, stable anonymization. At the same time, detection accuracy reached parity with the state-of-the-art, thanks to the track state machine that eliminated false “phantom” detections.

Detection accuracy

On the face detection side, our chosen detector performs strongly across the standard easy, medium, and hard benchmark subsets, as shown in Figure 4. Detection remains highly reliable on clearly visible faces, with the expected and well-understood difficulty arising only on the hardest cases — heavily occluded faces or faces smaller than a handful of pixels. In accordane with this, we have achieved a best-case scenario accuracy of >91% and a worst-case conditions accurarcy of >59% using WiderFace dataset (>32000 images).

Detection accuracy Figure 4: Face detection validation accuracy (AP)

The throughput trade-off

Our pipeline operates in real time on modest hardware. On our benchmark setup—featuring an NVIDIA GeForce MX550 GPU—it processes video at approximately 22 frames per second (FPS). The Python-based application utilizes OpenCV as the primary rendering library, Gstreamer as multimedia framework and ONNX Runtime as the core AI backend.

A more heavily engineered state-of-the-art baseline runs about twice as fast — but, importantly, that gap is an engineering advantage from batching and vectorization, not an algorithmic one. In other words, our tracking quality already exceeds the state-of-the-art, and closing the speed gap is a matter of well-understood optimization rather than fundamental redesign. That makes throughput our clearest and most achievable next milestone.

In practical terms, this level of accuracy and speed means the system can reliably anonymize people in real time while keeping the identities you choose visible — making it dependable for real-world broadcast, security, and privacy applications.

Conclusion

We are dedicated to advancing AI-driven solutions that address real-world multimedia challenges. Our on-the-edge face anonymization system is the result of extensive research, rigorous benchmarking, and deep expertise in AI, multimedia processing, and computer vision. By offering a fully offline, highly accurate, and selective solution, we empower businesses to anonymize, recognize, and manage faces in video efficiently — while ensuring data privacy, reducing operational costs, and supporting regulatory compliance.

Based on these results, we consider the current algorithm highly feasible for integration into our Fluendo AI Plugins. Thanks to its parallelizable, graph-based, fully GPU-driven architecture, FluPluginsAI will drastically reduce preprocessing and postprocessing timestamps. Furthermore, this architectural shift—combined with expected optimizations that will significantly reduce model inference time—makes real-time video processing at 4K resolution fully achievable.

Looking ahead, we remain committed to continuous innovation and improvement. Some key areas of future development include:

  • Latency optimization: Adopting batched inference and vectorized processing to close the throughput gap and push frame rates higher, enabling even smoother real-time performance.
  • Stronger privacy guarantees: Adding at-rest encryption for the identity gallery so that stored biometric data is protected by design.
  • Cloud & edge flexibility: Extending the solution to cloud-based deployments for greater scalability while preserving its fully offline capabilities.
  • Cutting-edge recognition: Exploring Visual Language Models (VLMs) and other emerging techniques to further improve recognition in the most difficult, unconstrained conditions.

With our expertise, commitment to innovation, and deep understanding of multimedia technologies, we are shaping the future of privacy-preserving video intelligence.

Are you ready to take your multimedia projects to the next level? Let’s make it happen! Contact us here and discover how Fluendo can help you bring your ideas to life. Together, we’ll continue shaping the future of multimedia.