EfficientNet Architecture

Since my last post on EfficientNet on LinkedIn I started further reading on building blocks of the EfficientNet architecture, it can be divided into Architecture (Mobile Inverted Residual Block, Squeeze and Excitation optimization) and Training (Swish activation / Quantization friendly hard Swish activation, Stochastic Depth with Survival probability etc).

EfficientNet architecture is very well explained recently by Aman Arora so if you dont want to go over the paper yourself you should at least read this and off course last year my friend Connor Shorten published a video explaining this paper.

The best explanations of Squeeze and Excitation network are given here and here and Inverted Residual Bottlenecks which were introduced in MobileNetV2 are explained here.

The Fused Convolution, Superpixels and Conditional Conv (Cond conv) are not mentioned in their paper but they are only found in the code implementation.

An implementation of CondConv is also available in Tensorflow Github repository.

There is also a mobile-friendly EfficientNet architecture – EfficientNet-Lite, which removes Squeeze and Excitation network, replaces Swish activation with Relu6 for supporting post-training quantization and heterogeneous hardware.

Criticism and improvements of EfficientNet Architecture:

Here’s an article How Efficient is EfficientNet? that says EfficientNet’s performance on small dataset leaves much to be desired and ML community on Reddit and Twitter seems to agree.

Also there is an article from late July A more parameter-efficient bottleneck for EfficientNet! that introduces Efficient Channel Attention ‘ECA’, it shows that if you replace EfficientNet Linear bottleneck Squeeze-Excitation ‘SE’ layer with Linear Bottleneck ECA it is slightly better in terms of accuracy and also reduces the number of parameters by 12%

Book Review – Raspberry Pi for Computer Vision

Adrian Rosebrock of PyImageSearch.com recently released this new book so in this post, I decided to review it. The book starts with an introduction to Raspberry Pi (RPi), Computer Vision and Deep Learning, with clear explanation of what’s changed from few years ago and why its now suitable to run Computer vision and Deep learning algorithms on RPi, what are co-processor devices Intel Movidius NCS 2, Coral USB Accelerator and how image processing, Computer vision and Deep learning overlap.

Each chapter starts with chapter Learning Objectives which you can read to understand the key takeaways from a chapter before going in the details. After the first few introductory chapters rest of the book is written in a way that you can pick a chapter which is close to the problem that you’re trying to solve and go from there such as if you’re looking to change USB camera parameter or work with multiple cameras with the RPi, this book has chapters on each of these topics and other such projects, full list of topics in this Hobbyist as well as upcoming Hacker and Complete bundles can be found on PyImageSearch.com so I’m not going to repeat them in this post.

Though this book is targeted to those who are new to Computer vision or Raspberry Pi but it is filled with information that is useful for even those with some experience in OpenCV and Computer Vision. Arguably some of the material in the book has been covered previously on PyImageSearch.com blog but Adrian expands on most of these in this book which allows much more room to include all the details unlike a blog post, so if you love his blog posts wait till you read his book.

The book has also helped me come up with a few tricks of my own when I was stuck at the same problem for a while but then I had an Ahaa Moment after I read Adrian’s book and it was interesting to get an insight into how Adrian and his team did a few revisions to arrive at the right solution given the limitations that exist on an RPi computer due to CPU resource and how to work around them. If you are starting out with OpenCV and Computer Vision for edge computing then you should definitely check out this book.

Meanwhile, I’m looking forward to getting Hacker and Complete Bundle when things get much more serious with the introduction of Deep Learning models, many more practical use cases and use of edge computing hardware like Intel Movidius NCS 2, Coral USB Accelerator and Nvidia Jetson Nano.

Review of Deep Learning for Computer Vision with Python – Practitioner Bundle

Deep Learning for Computer Vision - Practitioner Bundle Review

Some of you know that I have been reading Adrian Rosebrock’s book – Deep Learning for Computer Vision with PythonDL4CV”, I did a review of the Starter bundle a few months back. Recently I finished reading Practitioner Bundle so here’s a review of this book.

Practitioner bundle starts from where Starter bundle left off. While Starter bundle gives you the necessary introduction to the field of Computer vision and Image processing it’s more geared towards the beginners who are just entering the field of Deep learning for Computer vision, Practitioner Bundle is suited for more real-life uses cases. In this book, Adrian walks you through some of the practical tips and tricks of writing your own Convolution neural networks, training and evaluating them with real data sets and so on

First covered in the book is Data Augmentation technique for your training input data which is a recommended pre-processing step to have better generalization capability in your neural network during inference, Adrian shows you all the necessary steps to do Data augmentation before training any network.

One of the core techniques for enhancing your Neural network and getting accurate predictions is Transfer Learning. It is a technique where you use existing neural networks which are trained on similar data sets and modify them to suit your own training data, in the book Adrian shows you how to do that along with Fine-tuning a neural network.

Adrian also shows what is rank-1, rank-5 accuracy and when to use which one, how to use an ensemble of a neural network to get even higher accuracies, what are different network optimization methods like Adagrad, RMSProp, and Adam when to use each of them with examples.

Like the Starter bundle, this Practitioner bundle is structured in a way that every chapter builds on top of the previous one.  After the chapters on Image Recognition and Classification, Adrian takes the natural next step which is Object Detection along with other more recent researches in Convolutional Neural networks that resulted in Neural Style Transfers, Image Super-Resolution and Generative Adversarial Networks (GANs).

As explained in the Code section of my Starter Bundle blog post, the code only gets better in this bundle where every chapter not only has a code for it but also from the previous chapters which make it easier to get an understanding of how all of the concepts work together. The code is structured in such a way that it’s easier to read with descriptive comments and it’s even ready to be used in your own projects without modifications.Speaking ResNet

Personally, this book has helped me better understand architectures like ResNet, GoogleNet and how to implement them among other things. Adrian has done a really good job of explaining every detail of these architectures including code implementation in Keras, this helped me explain the ResNet architecture to professionals recently in a Computer Vision-based meetup in Karachi that I organized.

In the beginning I had to read several blog posts to learn each concept for example data augmentation, transfer learning or working with large datasets but I still couldn’t piece them together not to mention that code examples varied in quality, DL4CV and the code that accompanies the book has all of those necessary information and more in one place, it has given me that solid understanding that I have used in my own work.

I moderate the largest Deep Learning group on Facebook and see a lot of people who are new in the field of Computer vision struggle with some of these concepts and I often recommend this book to them.

I totally enjoyed reading this book and it’s a book which you’ll either have a copy sitting on your desk or just a click away in your favorite PDF reader on your computer when you’re implementing or training your own networks

Happy reading!

Review of Deep Learning for Computer Vision with Python – Starter Bundle

Deep Learning for Computer Vision with Python book

Last month I started reading through Adrian Rosebrock’s latest book Deep Learning for Computer Vision with Python, this book is divided in 3 bundles, Starter, Practitioner and Image Net bundle.

Each bundle is targeted at different audience, for those familiar with Python, Machine Learning and looking to get started with Deep Learning for computer vision there is Starter Bundle and data scientists looking to apply Image Recognition to their own problems can go for Practitioner and Researchers would be more interested in the ImageNet Bundle

Starter Bundle

I completed reading Starter Bundle recently so I decided to share my review in this post.

As oppose to some other books which assume prior knowledge of basics of deep learning with Convolutional Neural Networks and image processing, Adrian starts from the early days and history of deep learning explaining why it didn’t work and why now, then he goes to show the fundamentals of image processing and how they are constructed, this gives a solid foundation for rest of the book especially to newcomers in the field of image processing and computer vision.

Dividing his book in 3 different bundles allows him to expand on every bit of detail that is important, to give you an idea Starter bundle alone consists of 23 chapters, starting from basics to case studies where learners can apply their knowledge with practical code examples. Some other books for the sake of completeness and in favor of keeping low number of pages would generally avoid going in depth into those details.

Prior to reading this book I have learned a lot of this from various courses and blog posts so having seen those topics covered in such a detailed way in one place which builds on top of learnings from the previous chapter looked very refreshing to me

Adrian spends a good amount of time implementing a neural network from scratch and guides you along the way, this is different from other sources who would jump straight into Tensorflow or Keras without building an intuition of the reader. In my opinion if you are new to deep learning, implementing a Neural network yourself is the key to understanding the inner working before diving into frameworks which hide some of the low-level details from you


This book comes with code for each chapter that is not only detailed and easy to understand but if you are an experienced developer it is also ready to be used in your own implementations. Code is covered with very descriptive comments to help you understand what’s going on in every block of code which is very helpful when using it.

Also in his book Adrian highlights some of the subtle details such as the Keras configuration file, in my opinion most of the books would just skip over and go straight to coding. I find this helpful and if it wasn’t for this book I would not have considered this file for a long time, what those values are and how to change them when required, for example Theano uses channels first ordering whereas Tensorflow uses Channels last ordering when processing images so depending on which underlying framework you use for Keras you need to change this setting in the config file. All code examples in the book are also easier to run even in the cloud where I have tested them myself

I have definitely learned new techniques such as how to schedule your learning rate, techniques to spot underfitting and overfitting by babysitting your deep learning models and so on.

Like I covered earlier, this Starter bundle ends by showing you practical case-studies including obtaining and labelling datasets from scratch, training your models and prediction with live-camera or video.


I totally enjoyed reading this book and I can’t wait to start reading the Practitioner bundle and even implement some of the learning in my own work. If you’re new to deep learning and looking to get started I recommend that you read this book

Happy learning!

Thoughts on Course 1 of Prof. Andrew Ng’s DeepLearning.ai specialization

I recently passed my first course of the DeepLearning.ai specialization on Coursera and I decided to share some thoughts from someone new to Deep Learning.



Course 1 is structured very nicely, in week 1 it starts from the very basic of Neural Networks, though as the course prerequisite says a user needs some experience with Machine Learning ideally using Python to grasp concepts presented in these lectures.

In the following week basics of binary classification and logistic regression are explained including the cost function, gradient descent and derivatives, as well as basics of vectorization in python and shown with examples why vectorization is so important in Deep Learning.

I feel that there cannot be a simpler way to show this concept than this. Each lecture video in the course builds on the lecture before it which makes it easier to digest information in chunks or go over a particular concept multiple times to fully understand it.

In week 3 you implement a Shallow Neural Network with the knowledge you have gained from previous lectures. You learn that a Shallow Neural Network is a neural network with 1 hidden layer, in this week you build and use activation functions, vectorization, computing costs, gradient descent and more.

Finally in week 4 you learn to implement a fully connected deep neural network including forward and backward propagations, this week also gives difference of parameters and hyper parameters and why they are important, which is covered in Course 2


Quiz and Assignments are nicely prepared to help you gauge your own understanding, they cover all important concepts delivered throughout the lectures, I recommend spending some time there to fully understand those lectures before jumping to the following week.

I would highly recommend taking this course to anyone with experience in Machine Learning looking to start with Deep Learning.