Background Blurring with Semantic Image Segmentation using DeepLabv3+

Few days ago Microsoft released a version of Teams which uses AI to blur your background during video calls.

ezgif.com-optimize

Here’s a video on YouTube

This gave me an idea to try building this myself using AI. A few months ago Google open sourced DeepLab, a state of the art research for semantic image segmentation

From DeepLab Github

DeepLab: Deep Labelling for Semantic Image Segmentation

DeepLab is a state-of-art deep learning model for semantic image segmentation, where the goal is to assign semantic labels (e.g., person, dog, cat and so on) to every pixel in the input image.

So I used a Keras implementation of DeepLabv3+ to blur my background when I use my webcam

ezgif-3-8d13b793b6

In this post I’ll show you how to build this

image

After installing DeepLabV3+ lets import it along with required libraries

image

Then we create an instance of DeepLabv3 model, initialize our webcam stream, create a window using OpenCV where our result will be displayed and set a value for the background blur that is required

image

In the above code we initialize a loop and read every frame streaming from our webcam. We then resize the input to be 512px wide.
In the following lines a padding is added to each frame to resize it to 512×512 pixels required by DeepLabv3 model, please note that can be changed when initializing the model by specifying your own desired input size.
Another dimension is added to the input frame using numpy expand_dims before calling the predict function of the model.

A labels array containing indices of the maximum values is created after we remove single dimensions from the output result.

image

We then cleanup labels array by padding some values from the bottom of the image. We create a mask over all the black pixels in the output array. Then we resize the input frame from our webcam to match the size of our output labels for fitting our mask we just created
Then we blur this resized frame, in the original frame I replace the values of every dark pixel with values from our blurred frame and show the results. We exit while loop and close the stream after pressing ‘q’ to clean up OpenCV and streaming video frames objects

You can grab the code from my github and if you like it dont forget to give it a Star 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *