ResNet And DenseNet Implementation In Depth

In recent years, convolutional neural networks (CNNs) have revolutionized the field of computer vision, enabling machines to recognize patterns, classify images, and even generate artwork. Among the numerous advancements in CNN architectures, ResNet and DenseNet stand out due to their innovative designs and remarkable performance improvements. In this article, we will delve into the concepts behind of ResNet and DenseNet, exploring their architectures, underlying principles, and practical implementations.

CNNs are designed to automatically and adaptively learn spatial hierarchies of features from input images. They consist of several types of layers, including:

1. Convolutional Layers: These layers apply convolution operations to the input, capturing local patterns such as edges, textures, and shapes.
2. Pooling Layers: These layers reduce the spatial dimensions of the feature maps, which helps in making the representations invariant to small translations.
3. Fully Connected Layers: These layers are typically used at the end of the network to perform classification based on the learned features.

A detailed tutorial on CNNs is available here.

CNNs have achieved state-of-the-art results in various tasks, but as the depth of the network increases, training becomes challenging due to issues like vanishing gradients. This is where ResNet and DenseNet come into play.

Prerequisites:

1. Python, Numpy, Sklearn, Pandas and Matplotlib.
2. Familiarity with TensorFlow and Keras
3. Linear Algebra For Machine Learning.
4. Statistics And Probability Theory.
5. Solid Understanding Of CNNs

The Challenge of Training Deep Networks

Before the advent of ResNet or DenseNet, the primary issue with deep neural networks was the degradation problem. As layers were added, the performance of the network often plateaued or degraded, rather than improved. This was contrary to the intuitive expectation that deeper networks should perform better due to their higher capacity to learn complex representations.

This degradation was not due to overfitting but rather stemmed from the difficulties in optimizing deep networks. Specifically, as the depth increased, the gradients used in backpropagation either diminished (vanishing gradient problem) or grew exponentially (exploding gradient problem), making training unstable and slow.

Residual Networks (ResNet)

ResNet (Residual Network) architecture stands as a landmark development that has significantly advanced the performance and capabilities of neural networks. Introduced by Kaiming He et al. in their 2015 paper “Deep Residual Learning for Image Recognition,“

The core innovation of ResNet is the introduction of residual learning. Instead of learning the desired underlying mapping directly, ResNet learns the residual mapping. If the desired mapping is represented as H(x), ResNet reformulates it as H(x) = F(x) + x, where F(x) is the residual function that the network learns, and x is the identity input. This simple yet profound change helps in addressing the degradation problem.

Residual learning is implemented through shortcut (or skip) connections that bypass one or more layers. These shortcut connections add the input of a layer directly to the output of a deeper layer: y = F(x, {Wi​}) + x.

Here, x is the input, F(x,{Wi}) represents the residual mapping to be learned, and yis the output. The shortcut connection directly adds xto F(x, {Wi}), ensuring that the gradient can flow directly through the network, mitigating the vanishing gradient problem and allowing the training of much deeper networks.

The ResNet architecture is composed of several building blocks called residual blocks. These blocks are stacked to form very deep networks.

A typical ResNet implementation (taken from Amritesh’s project) looks something like this:

1. Input Layer (Input): Defines the input shape for the model.
2. Initial Convolution: Applies a single convolution layer to the input.
3. Residual Blocks: Stacks multiple residual blocks, grouped into three main stages with increasing filter sizes and downsampling applied between stages.
4. Average Pooling (AveragePooling2D): Reduces the spatial dimensions.
5. Fully Connected Layer (Dense): Applies a dense layer with softmax activation for classification.

Implementation

A detailed explanation on how to implement ResNet is discussed in the project here. Do check it out and implement it on your own.

DenseNet Architecture

DenseNet (Densely Connected Convolutional Networks) represents a significant innovation that addresses several limitations of previous architectures. Introduced by Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger in their 2016 paper “Densely Connected Convolutional Networks,” DenseNet has proven to be a powerful and efficient model for various image recognition tasks.

DenseNet introduces a novel connectivity pattern for convolutional networks to solve the problem mentioned in the beginning. Unlike traditional architectures where each layer has connections only to its immediate predecessor and successor, DenseNet connects each layer to every other layer (within the dense block) in a feed-forward fashion. This means that the input to each layer includes the feature maps of all preceding layers. This dense connectivity pattern aims to improve information flow and gradient propagation throughout the network.

The core building block of DenseNet is the dense block. In a dense block with L layers, each layer receives inputs from all previous layers and passes its own output to all subsequent layers. Mathematically, the input to the lth layer is the concatenation of the feature maps produced by all preceding layers:

xl = Hl([x0,x1,...,xl−1]) where x_l​ is the output of the lth layer, [x0,x1,...,xl−1] denotes the concatenation of feature maps from layers 0 to l−1, and Hl​ represents the transformation (e.g., a convolution operation) applied by the lth layer.

Between dense blocks, DenseNet incorporates transition layers that perform down-sampling. These layers consist of a batch normalization layer, a 1×1 convolutional layer (to reduce the number of feature maps), and a 2×2 average pooling layer. Transition layers help control the complexity and size of the network by reducing the number of feature maps.

The DenseNet architecture is characterized by its dense blocks and transition layers. It can be customized in terms of depth (number of layers) and growth rate (number of filters added per layer in a dense block). The growth rate, denoted as k, is a critical hyperparameter in DenseNet. It determines the number of filters added at each layer within a dense block. If the growth rate is set to 32, each layer will add 32 new feature maps. The total number of output feature maps after L layers in a dense block is k0+L⋅k, where k0​ is the number of input feature maps to the dense block.

The dense connectivity pattern ensures that each layer has direct access to the gradients from the loss function and the original input signal, leading to more effective feature reuse. This improves the learning process by enabling the network to learn richer and more diverse features. DenseNet’s architecture alleviates the vanishing gradient problem by allowing gradients to flow directly through the network via the dense connections. This results in better gradient propagation and more stable training, especially in very deep networks.

DenseNet is more parameter-efficient compared to traditional architectures. Due to the feature reuse, DenseNet can achieve comparable or even better performance with fewer parameters. This efficiency makes DenseNet attractive for applications with limited computational resources. The enhanced feature reuse and parameter efficiency contribute to reduced overfitting, particularly on smaller datasets. DenseNet’s ability to learn diverse features from the same input data helps in building more generalizable models.

DenseNet-BC is a variant of DenseNet that incorporates two modifications:

1. Bottleneck Layers: Introduces 1×1 convolutions before the 3×3 convolutions within dense blocks, reducing the number of input feature maps and thus computational cost.
2. Compression: Uses compression in transition layers by reducing the number of feature maps by a factor of θ (typically set to 0.5).

These modifications further improve the parameter efficiency and performance of DenseNet.

Implementation

A detailed implementation is shown and discussed here in the project by Amritesh.

Final Note:

Both ResNet and DenseNet have significantly advanced the field of deep learning through their innovative architectures, addressing key challenges in training deep neural networks.

ResNet introduced residual learning with shortcut connections, mitigating the vanishing gradient problem and enabling the training of deeper networks with enhanced performance. DenseNet, on the other hand, leveraged dense connections to improve feature reuse, gradient propagation, and parameter efficiency, resulting in more effective learning and reduced overfitting.

Together, these architectures have set new benchmarks in various applications, including image recognition, medical imaging, and more, showcasing their versatility and robustness. Their contributions continue to inspire further innovations, pushing the boundaries of deep learning capabilities.