Katkat
Back

Explain CNN shapes, params, and trade-offs

AppleData Scientist · Onsite
MediumData ScientistOnsiteMachine Learning

Last updated

This question tests your ability to calculate output dimensions, parameter counts, and computational costs (FLOPs/MACs) in CNNs, while requiring you to justify architectural choices like depthwise convolutions and normalization layers.

The Problem

Consider a standard 2D convolutional layer. You are given the following input tensor dimensions and kernel specifications:

  • Input image size: H x W x C (Height, Width, Channels)
  • Number of filters (kernels): K
  • Kernel spatial size: f x f
  • Stride: s
  • Padding: p (assume zero-padding)

Part 1: Dimensionality and Parameters

  1. Derive the formula for the spatial output dimensions H_out and W_out.
  2. Determine the total number of trainable parameters in this convolutional layer (including bias).
  3. Calculate the number of multiply–accumulate (MAC) operations required for a single forward pass through this layer.

Part 2: Architectural Trade-offs

Suppose you replace this standard convolution with a depthwise separable convolution (as seen in MobileNets) and compare the parameter count and computation cost relative to the standard version.

Part 3: Normalization

Explain the trade-offs of placing Batch Normalization before versus after the activation function (e.g., ReLU). What happens if the network is extremely deep or the batch size is very small (e.g., N=1)?

Example

Assume an input of 32 x 32 x 3 (RGB image), with K=16 filters, f=3, s=1, and p=1 (padding to maintain spatial size).

  • H_out = 32, W_out = 32
  • Parameters = (f * f * C + 1) * K = (3 * 3 * 3 + 1) * 16 = 448
  • MACs ≈ H_out * W_out * K * f * f * C = 32 * 32 * 16 * 3 * 3 * 3 = 442,368

Hints

Hint 1

Recall that the output spatial dimension is `floor((Input + 2*padding - kernel_size) / stride) + 1`.

Hint 2

For depthwise separable convolutions, split the task into a depthwise convolution (one filter per channel) and a pointwise convolution (1x1 kernel).

Hint 3

Batch normalization depends on mini-batch statistics; if the batch size is 1, the variance is zero and the mean is just the input itself, rendering the transformation constant.

Follow-up questions

  • How does the use of dilated (atrous) convolutions affect the receptive field and the parameter count?
  • If we move from Batch Normalization to Layer Normalization or Group Normalization, how does the dependence on batch size change?
  • What are the implications for hardware acceleration (e.g., GPU/TPU) when using depthwise separable convolutions versus standard convolutions?