Chapter 2. Deep Neural Networks

In this chapter, we'll be examining deep neural networks. These networks have shown excellent performance in terms of the accuracy of their classification on more challenging and advanced datasets like ImageNet, CIFAR10, and CIFAR100. For conciseness, we'll only be focusing on two networks, ResNet [2][4] and DenseNet [5]. While we will go into much more detail, it's important to take a minute to introduce these networks:

ResNet introduced the concept of residual learning which enabled it to build very deep networks by addressing the vanishing gradient problem in deep convolutional networks.

DenseNet improved the ResNet technique further by allowing every convolution to have direct access to inputs, and lower layer feature maps. It's also managed to keep the number of parameters low in deep networks by utilizing both the Bottleneck and Transition layers.

But why these two models, and not others? Well, since their introduction, there have been countless models such as ResNeXt [6] and FractalNet [7] which have been inspired by the technique used by these two networks. Likewise, with an understanding of both ResNet and DenseNet, we'll be able to use their design guidelines to build our own models. By using transfer learning, this will also allow us to take advantage of pretrained ResNet and DenseNet models for our own purposes. These reasons alone, along with their compatibility with Keras, make the two models ideal for exploring and complimenting the advanced deep learning scope of this book.

While this chapter's focus is on deep neural networks; we'll begin this chapter by discussing an important feature of Keras called the Functional API. This API acts as an alternative method for building networks in Keras and enables us to build more complex networks that cannot be accomplished by the sequential model. The reason why we're focusing so much on this API is that it will become a very useful tool for building deep networks such as the two we're focusing on in this chapter. It's recommended that you've completed, Chapter 1, Introducing Advanced Deep Learning with Keras, before moving onto this chapter as we'll refer to introductory level code and concepts explored in that chapter as we take them to an advanced level in this chapter.

The goals of this chapter is to introduce: