Deep Learning, CNNs & Transformer Architectures

Welcome to the definitive guide to Deep Learning, Computer Vision, Sequence Modeling, and Transformer Architectures. Modern deep networks are grounded in deep linear algebra, multivariable calculus, and variance-preserving statistical dynamics.

This curriculum derives every fundamental deep learning component from mathematical first principles and implements each architecture from scratch in both pure Python and pure Rust without relying on PyTorch, TensorFlow, or external numerical libraries.

Explore lessons from the sidebar to inspect step-by-step mathematical proofs, zero-dependency implementations, interactive zoomable diagrams, and knowledge verification checkpoints.

Curriculum Architecture at a Glance#

This course provides comprehensive mathematical foundations and production-grade implementations across the deep learning lifecycle:

1. Numerical Stability & Normalization Layers: Variance preservation proofs for Xavier/Glorot and He/Kaiming initializations, Batch Normalization mini-batch moment statistics, and Layer Normalization channel-wise invariance without batch dependencies.
2. Convolutional Networks & Residual Highways: Discrete 2D cross-correlation spatial transformations, receptive field dilation mathematics, ResNet identity mapping formulations ($y = F(x, W_l) + x$), and mathematical proofs of uninterrupted gradient backpropagation.
3. Recurrent Dynamics & Gating Circuits: Vanishing and exploding gradient analyses in recurrent transitions, Long Short-Term Memory (LSTM) cell state additive updates and gating mechanics, and Gated Recurrent Unit (GRU) parameter compression.
4. Attention Mechanisms & Transformer Foundations: Scaled Dot-Product Attention derivation ($\text{Softmax}(QK^T / \sqrt{d_k})V$), variance scaling proofs, Multi-Head projection mechanics, sinusoidal positional encodings, and rotary position embeddings (RoPE).
5. Parameter-Efficient Fine-Tuning (PEFT & LoRA): Intrinsic dimension hypothesis, Low-Rank Adaptation weight reparameterization ($W = W_0 + \frac{\alpha}{r}BA$), rank-$r$ decomposition scaling, and memory-efficient training dynamics.
6. Graph Neural Networks & Spectral Message Passing: Normalized graph Laplacians ($L = I - D^{-1/2} A D^{-1/2}$), spectral graph convolutions, Kipf-Welling first-order localized approximations, and spatial permutation-invariant message-passing paradigms.

Core Pedagogical Standard#

• Step-by-Step Algebraic Derivations: Every tensor contraction, matrix Jacobian, and backpropagation chain is derived algebraically step by step.
• Zero-Dependency Implementations: Every layer and model architecture includes complete, runnable implementations in pure Python (standard library only) and pure Rust (std only).
• Interactive Visual Architecture: Explore interactive flowcharts, mathematical formulas rendered with KaTeX, expandable diagrams, and knowledge verification checkpoints.

The complete course source code and implementations are open on the Source Repository.