Deep Learning for Finance

01 — Fundamentals

Machine learning and deep learning: Representation learning, inductive bias, empirical risk minimization.
Neural network architecture: Multi-layer perceptrons, matrix computations, computation graphs.
Weights, biases and activation functions: Non-linear activations (ReLU, Sigmoid, Tanh, Softplus, GELU) and parameter initialization.
Forward propagation and loss: Binary Cross-Entropy, Mean Squared Error, and probabilistic likelihoods.
Gradients, backpropagation and training: Exact multivariable chain rule, Jacobian-vector products, SGD, and adaptive optimizers (Adam).

02 — Applications in finance

Fraud detection: Extreme class imbalance, anomaly detection, precision-recall optimization, and cost-sensitive loss.
Credit risk and loan default prediction: Calibrated probability of default (PD), expected loss modeling, and interpretable credit scorecards.
Financial time-series forecasting: Volatility clustering, non-stationarity, market regime shifts, and multi-horizon returns.
Neural networks, LSTMs and Transformers: Recurrent dynamics, Long Short-Term Memory gates, self-attention, and sequence modeling.
Building blocks of a financial AI system: Robust latency pipelines, strict point-in-time backtesting, and risk-management boundaries.

Mathematical Exercises

These four mathematical exercises develop rigorous analytical proficiency in neural network mechanics applied to financial problems. Expand the solution dropdown in each exercise to inspect complete mathematical derivations.

Exercise 1: Analytical Forward & Backpropagation for Credit Risk Classification

Calculus & Optimization

Financial Context: In consumer credit risk underwriting, an institution predicts loan default ($y = 1$) versus repayment ($y = 0$) using financial applicant covariates.

Model Formulation:

Consider a 2-layer Multi-Layer Perceptron (MLP) with:

Input $x = [x_1, x_2]^T \in \mathbb{R}^2$ where $x_1$ is Debt-to-Income (DTI) and $x_2$ is revolving credit utilization.
Hidden layer with 2 neurons and ReLU activation $\operatorname{ReLU}(z) = \max(0, z)$: $$z^{[1]} = W^{[1]} x + b^{[1]}, \quad a^{[1]} = \operatorname{ReLU}(z^{[1]})$$ $$W^{[1]} = \begin{pmatrix} 0.5 & -0.2 \\ 0.8 & 0.4 \end{pmatrix}, \quad b^{[1]} = \begin{pmatrix} 0.1 \\ -0.3 \end{pmatrix}$$
Output layer with 1 neuron and Sigmoid activation $\sigma(z) = \frac{1}{1 + e^{-z}}$ predicting default probability $\hat{y}$: $$z^{[2]} = W^{[2]} a^{[1]} + b^{[2]}, \quad \hat{y} = \sigma(z^{[2]})$$ $$W^{[2]} = \begin{pmatrix} 1.2 & -0.9 \end{pmatrix}, \quad b^{[2]} = -0.1$$
Loss function: Binary Cross-Entropy $\mathcal{L}(y, \hat{y}) = -[y \ln(\hat{y}) + (1-y)\ln(1-\hat{y})]$.

Questions:

For applicant $x = [0.6, 0.8]^T$ with true label $y = 1$ (default), calculate $z^{[1]}$, $a^{[1]}$, $z^{[2]}$, the default probability $\hat{y}$, and loss $\mathcal{L}$.
Prove analytically that $\frac{\partial \mathcal{L}}{\partial z^{[2]}} = \hat{y} - y$.
Compute exact numerical gradients $\frac{\partial \mathcal{L}}{\partial W^{[2]}}$, $\frac{\partial \mathcal{L}}{\partial b^{[2]}}$, hidden error $\delta^{[1]} = \frac{\partial \mathcal{L}}{\partial z^{[1]}}$, $\frac{\partial \mathcal{L}}{\partial W^{[1]}}$, and $\frac{\partial \mathcal{L}}{\partial b^{[1]}}$.
Compute updated parameters following one gradient descent step with learning rate $\eta = 0.1$.

View Step-by-Step Mathematical Solution

1. Forward Pass:

$$z_1^{[1]} = 0.5(0.6) + (-0.2)(0.8) + 0.1 = 0.30 - 0.16 + 0.10 = 0.24$$ $$z_2^{[1]} = 0.8(0.6) + 0.4(0.8) - 0.30 = 0.48 + 0.32 - 0.30 = 0.50$$ $$a_1^{[1]} = \max(0, 0.24) = 0.24, \quad a_2^{[1]} = \max(0, 0.50) = 0.50$$ $$z^{[2]} = 1.2(0.24) + (-0.9)(0.50) - 0.1 = 0.288 - 0.450 - 0.100 = -0.262$$ $$\hat{y} = \sigma(-0.262) = \frac{1}{1 + e^{0.262}} \approx \frac{1}{1 + 1.2995} \approx \mathbf{0.4349}$$ $$\mathcal{L}(1, 0.4349) = -\ln(0.4349) \approx \mathbf{0.8327}$$

2. Output Gradient Derivation:

$$\frac{\partial \mathcal{L}}{\partial \hat{y}} = -\frac{y}{\hat{y}} + \frac{1-y}{1-\hat{y}} = \frac{\hat{y} - y}{\hat{y}(1-\hat{y})}, \quad \frac{\partial \hat{y}}{\partial z^{[2]}} = \hat{y}(1-\hat{y})$$ $$\frac{\partial \mathcal{L}}{\partial z^{[2]}} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z^{[2]}} = \frac{\hat{y}-y}{\hat{y}(1-\hat{y})} \cdot \hat{y}(1-\hat{y}) = \mathbf{\hat{y} - y} \quad \blacksquare$$

3. Exact Numerical Gradients:

$$\delta^{[2]} = \frac{\partial \mathcal{L}}{\partial z^{[2]}} = 0.4349 - 1 = \mathbf{-0.5651}$$ $$\frac{\partial \mathcal{L}}{\partial W^{[2]}} = \delta^{[2]} (a^{[1]})^T = -0.5651 \times [0.24, 0.50] = \mathbf{[-0.1356, \; -0.2826]}$$ $$\frac{\partial \mathcal{L}}{\partial b^{[2]}} = \delta^{[2]} = \mathbf{-0.5651}$$ $$\delta^{[1]} = ((W^{[2]})^T \delta^{[2]}) \odot \mathbf{1}_{z^{[1]}>0} = \begin{pmatrix} 1.2 \\ -0.9 \end{pmatrix} (-0.5651) \odot \begin{pmatrix} 1 \\ 1 \end{pmatrix} = \mathbf{\begin{pmatrix} -0.6782 \\ 0.5086 \end{pmatrix}}$$ $$\frac{\partial \mathcal{L}}{\partial W^{[1]}} = \delta^{[1]} x^T = \begin{pmatrix} -0.6782 \\ 0.5086 \end{pmatrix} \begin{pmatrix} 0.6 & 0.8 \end{pmatrix} = \mathbf{\begin{pmatrix} -0.4069 & -0.5425 \\ 0.3052 & 0.4069 \end{pmatrix}}$$ $$\frac{\partial \mathcal{L}}{\partial b^{[1]}} = \delta^{[1]} = \mathbf{\begin{pmatrix} -0.6782 \\ 0.5086 \end{pmatrix}}$$

4. Gradient Descent Updates ($\eta = 0.1$):

$$W^{[2]\prime} = [1.2, -0.9] - 0.1[-0.1356, -0.2826] = \mathbf{[1.2136, \; -0.8717]}$$ $$b^{[2]\prime} = -0.1 - 0.1(-0.5651) = \mathbf{-0.0435}$$ $$W^{[1]\prime} = \begin{pmatrix} 0.5 & -0.2 \\ 0.8 & 0.4 \end{pmatrix} - 0.1 \begin{pmatrix} -0.4069 & -0.5425 \\ 0.3052 & 0.4069 \end{pmatrix} = \mathbf{\begin{pmatrix} 0.5407 & -0.1458 \\ 0.7695 & 0.3593 \end{pmatrix}}$$ $$b^{[1]\prime} = \begin{pmatrix} 0.1 \\ -0.3 \end{pmatrix} - 0.1 \begin{pmatrix} -0.6782 \\ 0.5086 \end{pmatrix} = \mathbf{\begin{pmatrix} 0.1678 \\ -0.3509 \end{pmatrix}}$$

Exercise 2: Recurrent Dynamics, Vanishing Gradients, and Spectral Norms

Time-Series & Linear Algebra

Financial Context: In asset volatility prediction and market regime classification, financial sequences span hundreds of trading days ($T \ge 252$). Standard recurrent models often fail to retain long-memory trends.

Model Formulation:

A standard Elman Recurrent Neural Network updates hidden states via:

$$h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h), \quad t = 1, \dots, T$$

where $h_t \in \mathbb{R}^m$, $W_{hh} \in \mathbb{R}^{m \times m}$, and a loss $\mathcal{L} = \ell(h_T)$ is evaluated at horizon $T$.

Questions:

Show using the multivariable chain rule that $\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \frac{\partial h_k}{\partial h_{k-1}}$ and derive the single-step Jacobian $\frac{\partial h_k}{\partial h_{k-1}}$.
Using the induced matrix 2-norm (spectral norm $\|A\|_2 = \sigma_{\max}(A)$), prove the upper bound: $$\left\| \frac{\partial h_T}{\partial h_t} \right\|_2 \le \left( \sigma_{\max}(W_{hh}) \right)^{T - t}$$
Contrast the asymptotic behavior as $(T - t) \to \infty$ for $\sigma_{\max}(W_{hh}) < 1$ versus $\sigma_{\min}(W_{hh}) > 1$. Why does this explain why standard RNNs fail on low-frequency macroeconomic signals?
In an LSTM cell state $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$, evaluate $\frac{\partial c_t}{\partial c_{t-1}}$ and explain how initializing forget-gate bias $b_f \approx 1$ creates a "Constant Error Carousel" ($\frac{\partial c_T}{\partial c_t} \approx I$).

View Step-by-Step Mathematical Solution

1. Temporal Jacobian:

$$\frac{\partial h_T}{\partial h_t} = \frac{\partial h_T}{\partial h_{T-1}} \frac{\partial h_{T-1}}{\partial h_{T-2}} \cdots \frac{\partial h_{t+1}}{\partial h_t} = \prod_{k=t+1}^T \frac{\partial h_k}{\partial h_{k-1}}$$ Let $z_k = W_{hh} h_{k-1} + W_{xh} x_k + b_h$, so $h_k = \tanh(z_k)$. Since $\frac{d\tanh(u)}{du} = 1 - \tanh^2(u)$: $$\frac{\partial h_k}{\partial h_{k-1}} = \operatorname{diag}\left(1 - h_k^2\right) W_{hh}^T$$

2. Spectral Norm Bound Proof:

By sub-multiplicativity of induced matrix norms $\|AB\|_2 \le \|A\|_2 \|B\|_2$: $$\left\| \frac{\partial h_T}{\partial h_t} \right\|_2 \le \prod_{k=t+1}^T \left\| \operatorname{diag}\left(1 - h_k^2\right) W_{hh}^T \right\|_2 \le \prod_{k=t+1}^T \left\| \operatorname{diag}\left(1 - h_k^2\right) \right\|_2 \|W_{hh}^T\|_2$$ Since $\max_i |1 - \tanh^2(z_{k,i})| \le 1$ and $\|W_{hh}^T\|_2 = \sigma_{\max}(W_{hh})$: $$\left\| \frac{\partial h_T}{\partial h_t} \right\|_2 \le \prod_{k=t+1}^T 1 \cdot \sigma_{\max}(W_{hh}) = \mathbf{\left( \sigma_{\max}(W_{hh}) \right)^{T - t}} \quad \blacksquare$$

3. Financial Asymptotics:

Vanishing Gradient: When $\sigma_{\max}(W_{hh}) < 1$, as $(T - t) \to \infty$, the sensitivity bounds decay exponentially: $(\sigma_{\max})^{T-t} \to 0$. Macroeconomic regime signals occurring 60 or 252 trading days prior exert zero gradient influence on parameter updates.
Exploding Gradient: When $\sigma_{\min}(W_{hh}) > 1$ and states operate in linear regions, gradients explode exponentially: $\sigma_{\min}^{T-t} \to \infty$, destabilizing weights during high-volatility financial shocks.

4. LSTM Additive Error Highway:

$$\frac{\partial c_t}{\partial c_{t-1}} = \operatorname{diag}(f_t) + \left( \frac{\partial f_t}{\partial c_{t-1}} \odot c_{t-1} + \frac{\partial i_t}{\partial c_{t-1}} \odot \tilde{c}_t + \frac{\partial \tilde{c}_t}{\partial c_{t-1}} \odot i_t \right)$$ When forget gate bias $b_f \ge 1$, the sigmoid produces $f_t = \sigma(z_f) \approx 1$. Hence: $$\frac{\partial c_T}{\partial c_t} \approx \prod_{k=t+1}^T \operatorname{diag}(f_k) \approx \prod_{k=t+1}^T I = \mathbf{I}$$ The gradient flows linearly through the additive cell state across hundreds of market periods without geometric decay.

Exercise 3: Loss Curvature, Hessian Spectrum, and Learning Rate Bounds

Convex Optimization & Portfolios

Financial Context: In multi-factor equity models, cross-sectional excess asset returns $y \in \mathbb{R}^N$ are regressed on $d$ factor exposures $X \in \mathbb{R}^{N \times d}$ (Value, Momentum, Quality, Size, Beta).

Model Formulation:

A regularized neural layer with weights $w \in \mathbb{R}^d$ minimizes Mean Squared Error with $L_2$ weight decay:

$$\mathcal{L}(w) = \frac{1}{2N} \| y - X w \|_2^2 + \frac{\lambda}{2} \| w \|_2^2, \quad \lambda > 0$$

Questions:

Compute the analytical gradient $\nabla_w \mathcal{L}(w)$ and the Hessian matrix $H = \nabla^2_w \mathcal{L}(w)$.
Prove that $H$ is strictly positive definite for any $\lambda > 0$, even when factors in $X$ are highly collinear.
Under gradient descent $w^{(k+1)} = w^{(k)} - \eta \nabla \mathcal{L}(w^{(k)})$, prove that the error $e^{(k)} = w^{(k)} - w^*$ converges to zero if and only if: $$0 < \eta < \frac{2}{\lambda_{\max}(H)}$$
Prove that the optimal learning rate is $\eta^* = \frac{2}{\lambda_{\max}(H) + \lambda_{\min}(H)}$, and derive the optimal convergence rate $\rho^* = \frac{\kappa - 1}{\kappa + 1}$ where $\kappa = \frac{\lambda_{\max}(H)}{\lambda_{\min}(H)}$. Explain how weight decay $\lambda$ fixes ill-conditioned factor covariance matrices.

View Step-by-Step Mathematical Solution

1. Gradient and Hessian:

$$\mathcal{L}(w) = \frac{1}{2N} \left( y^Ty - 2 y^T X w + w^T X^T X w \right) + \frac{\lambda}{2} w^T w$$ $$\nabla_w \mathcal{L}(w) = \left( \frac{1}{N} X^T X + \lambda I \right) w - \frac{1}{N} X^T y$$ $$H = \nabla^2_w \mathcal{L}(w) = \mathbf{\frac{1}{N} X^T X + \lambda I} \quad \blacksquare$$

2. Positive Definiteness:

For any non-zero vector $v \ne 0$: $$v^T H v = \frac{1}{N} \|X v\|_2^2 + \lambda \|v\|_2^2 \ge 0 + \lambda \|v\|_2^2 > 0 \quad (\text{since } \lambda > 0)$$ All eigenvalues satisfy $\lambda_i(H) \ge \lambda > 0$. The loss surface is strictly convex with a unique global minimizer $w^* = (X^T X + N \lambda I)^{-1} X^T y$.

3. Maximum Stable Learning Rate:

Since $\nabla \mathcal{L}(w^*) = 0$, we have $\nabla \mathcal{L}(w^{(k)}) = H (w^{(k)} - w^*)$. Subtracting $w^*$: $$e^{(k+1)} = (I - \eta H) e^{(k)} \implies e^{(k)} = (I - \eta H)^k e^{(0)}$$ Convergence requires all eigenvalues of $(I - \eta H)$ to satisfy $|1 - \eta \lambda_i(H)| < 1$: $$-1 < 1 - \eta \lambda_i(H) < 1 \iff 0 < \eta < \frac{2}{\lambda_i(H)} \quad \forall i$$ Restricted by the largest eigenvalue: $\mathbf{0 < \eta < \frac{2}{\lambda_{\max}(H)}} \quad \blacksquare$

4. Optimal Rate and Conditioning:

The maximum contraction factor $\max \{ |1 - \eta \lambda_{\min}|, |1 - \eta \lambda_{\max}| \}$ is minimized when: $$1 - \eta^* \lambda_{\min}(H) = -(1 - \eta^* \lambda_{\max}(H)) \implies \mathbf{\eta^* = \frac{2}{\lambda_{\max}(H) + \lambda_{\min}(H)}}$$ $$\rho^* = \frac{\lambda_{\max}(H) - \lambda_{\min}(H)}{\lambda_{\max}(H) + \lambda_{\min}(H)} = \mathbf{\frac{\kappa - 1}{\kappa + 1}}, \quad \text{where } \kappa = \frac{\lambda_{\max}(H)}{\lambda_{\min}(H)}$$ In financial factor modeling, collinear factors cause $\lambda_{\min}(\frac{1}{N} X^T X) \approx 0$, leading to $\kappa \to \infty$ and crippling gradient descent. Adding $L_2$ regularization shifts the spectrum uniformly: $\kappa_{\text{reg}} = \frac{\lambda_{\max} + \lambda}{\lambda_{\min} + \lambda} \ll \infty$, dramatically accelerating optimization.

Exercise 4: Physics-Informed & Non-Arbitrage Neural Networks

Derivatives & PINNs

Financial Context: Neural networks trained to price European call option surfaces $C(S, K, \tau)$ must enforce economic no-arbitrage laws and satisfy the Black-Scholes partial differential equation (PDE).

Model Formulation:

Consider a neural pricing function $\hat{C}_\theta(S, K, \tau)$ using the Softplus activation $g(z) = \ln(1 + e^z)$:

$$\hat{C}_\theta(S, K, \tau) = \sum_{j=1}^m v_j g(w_j S + u_j K + p_j \tau + b_j) + b_0$$

Questions:

Prove that constraining outer weights $v_j \ge 0$ guarantees strike convexity $\frac{\partial^2 \hat{C}_\theta}{\partial K^2} \ge 0$, eliminating butterfly arbitrage and ensuring non-negative Breeden-Litzenberger risk-neutral probabilities $q(K) = e^{r\tau} \frac{\partial^2 C}{\partial K^2} \ge 0$.
Prove that $g'(z) = \sigma(z)$ and $g''(z) = \sigma(z)(1 - \sigma(z))$. Derive analytical closed forms for Delta $\hat{\Delta} = \frac{\partial \hat{C}}{\partial S}$ and Gamma $\hat{\Gamma} = \frac{\partial^2 \hat{C}}{\partial S^2}$.
Formulate the total Physics-Informed Neural Network (PINN) loss function balancing market observation data and the Black-Scholes PDE residual: $$\mathcal{N}[C] = \frac{\partial C}{\partial \tau} - \frac{1}{2} \sigma^2 S^2 \frac{\partial^2 C}{\partial S^2} - r S \frac{\partial C}{\partial S} + r C = 0$$
For a single neuron ($m=1$) with $v_1 = 1.0, w_1 = 0.5, u_1 = -0.5, p_1 = 0.2, b_1 = 0, b_0 = 0$, evaluate $\hat{C}$, $\hat{\Delta}$, $\hat{\Gamma}$, and the numerical PDE residual $\mathcal{N}[\hat{C}]$ at spot $S = 100$, strike $K = 100$, maturity $\tau = 1$, volatility $\sigma = 0.20$, and risk-free rate $r = 0.05$.

View Step-by-Step Mathematical Solution

1. Proof of Arbitrage-Free Convexity:

$$\frac{\partial \hat{C}_\theta}{\partial K} = \sum_{j=1}^m v_j u_j g'(z_j) \implies \frac{\partial^2 \hat{C}_\theta}{\partial K^2} = \sum_{j=1}^m v_j u_j^2 g''(z_j)$$ For Softplus, $g''(z) = \sigma(z)(1 - \sigma(z)) > 0$ for all $z \in \mathbb{R}$. Since $u_j^2 \ge 0$, if $v_j \ge 0$: $$v_j u_j^2 g''(z_j) \ge 0 \implies \mathbf{\frac{\partial^2 \hat{C}_\theta}{\partial K^2} \ge 0 \quad \forall K \in \mathbb{R}} \quad \blacksquare$$

2. Derivatives and Greeks:

$$g'(z) = \frac{e^z}{1 + e^z} = \sigma(z), \quad g''(z) = \sigma(z)(1 - \sigma(z))$$ $$\mathbf{\hat{\Delta} = \frac{\partial \hat{C}}{\partial S} = \sum_{j=1}^m v_j w_j \sigma(z_j)}$$ $$\mathbf{\hat{\Gamma} = \frac{\partial^2 \hat{C}}{\partial S^2} = \sum_{j=1}^m v_j w_j^2 \sigma(z_j)(1 - \sigma(z_j))}$$ When $v_j \ge 0$ and $w_j \ge 0$, $\hat{\Delta} \ge 0$ (monotonicity with spot) and $\hat{\Gamma} \ge 0$ (non-negative gamma).

3. Physics-Informed Loss Function:

$$\mathcal{L}_{\text{PINN}}(\theta) = \frac{1}{N}\sum_{i=1}^N \left(\hat{C}_\theta(S_i, K_i, \tau_i) - C_i^{\text{mkt}}\right)^2 + \frac{\lambda_{\text{PDE}}}{M}\sum_{k=1}^M \left( \frac{\partial \hat{C}_\theta}{\partial \tau_k} - \frac{1}{2}\sigma^2 S_k^2 \frac{\partial^2 \hat{C}_\theta}{\partial S_k^2} - r S_k \frac{\partial \hat{C}_\theta}{\partial S_k} + r \hat{C}_\theta \right)^2$$

4. Concrete Numerical Evaluation:

$$z_1 = 0.5(100) - 0.5(100) + 0.2(1) + 0 = \mathbf{0.2}$$ $$\hat{C} = \ln(1 + e^{0.2}) = \ln(1 + 1.2214) = \ln(2.2214) \approx \mathbf{0.7981}$$ $$\sigma(0.2) = \frac{1}{1 + e^{-0.2}} \approx \mathbf{0.5498}, \quad g''(0.2) = 0.5498(1 - 0.5498) \approx \mathbf{0.2475}$$ $$\frac{\partial \hat{C}}{\partial \tau} = (1.0)(0.2)(0.5498) = \mathbf{0.1100}$$ $$\hat{\Delta} = \frac{\partial \hat{C}}{\partial S} = (1.0)(0.5)(0.5498) = \mathbf{0.2749}$$ $$\hat{\Gamma} = \frac{\partial^2 \hat{C}}{\partial S^2} = (1.0)(0.25)(0.2475) = \mathbf{0.0619}$$ $$\text{Diffusion term: } \frac{1}{2} \sigma^2 S^2 \hat{\Gamma} = \frac{1}{2}(0.04)(10000)(0.0619) = 200 \times 0.0619 = \mathbf{12.38}$$ $$\text{Drift term: } r S \hat{\Delta} = 0.05(100)(0.2749) = \mathbf{1.3746}$$ $$\text{Discount term: } r \hat{C} = 0.05(0.7981) = \mathbf{0.0399}$$ $$\mathcal{N}[\hat{C}] = 0.1100 - 12.38 - 1.3746 + 0.0399 = \mathbf{-13.6047}$$

Curated Deep Learning Resources

Core foundational courses, lecture series, implementations, and theoretical treatises recommended for this course.

Neural Networks: Zero to Hero & Stanford CS231n

Andrej Karpathy — Former Director of AI at Tesla, OpenAI Founding Member

An indispensable, hands-on journey from microscopic foundations to state-of-the-art Transformer models. Karpathy walks through building neural network autograd engines from first principles, writing manual backpropagation kernels, designing language models, and training GPT models.

micrograd: A tiny autograd engine implementing scalar backpropagation.
makemore: Autoregressive generative modeling from Bigrams to MLPs, BatchNorm, and WaveNet.
nanoGPT: The cleanest, fastest repository for training medium-sized GPTs.
CS231n Notes: Iconic Stanford course notes on vector calculus, computational graphs, and neural optimization.

Zero to Hero Course YouTube Playlist CS231n Optimization Notes micrograd Repo A Recipe for Training NNs

Stanford CS230: Deep Learning & Deep Learning Specialization

Andrew Ng — Stanford Adjunct Professor, Co-founder of Coursera & DeepLearning.AI

Stanford University's CS230 and the Deep Learning Specialization provide the gold-standard systematic framework for deep learning. Andrew Ng covers deep network architectures, hyperparameter tuning, orthogonalization, gradient checking, convolutional systems, and sequence models (RNNs/LSTMs/Transformers).

CS230 Stanford: Applied deep learning methodology, error analysis, and project execution.
Deep Learning Specialization: Comprehensive 5-course curriculum on neural network intuition and math.
Stanford CS229: Rigorous statistical learning theory, optimization, and generalized linear models.

Stanford CS230 Website CS230 YouTube Lectures Deep Learning Specialization CS230 Super Cheatsheets Stanford CS229

The Principles of Deep Learning Theory

Daniel A. Roberts, Sho Yaida, and Boris Hanin — Cambridge University Press

A definitive, groundbreaking mathematical textbook treating deep neural networks through the lens of effective field theory and theoretical physics. Rather than viewing deep learning as a mysterious black box, this book constructs a systematic, first-principles microscopic theory of signal propagation through depth, critical weight-and-bias initialization, and finite-width corrections ($1/N$ expansion) that reveal why deep neural representations learn and generalize.

Effective Field Theory Framework: Unifies infinite-width Gaussian processes with realistic finite-width feature learning.
Signal Propagation & Criticality: Exact criteria for preserving variance and gradient information across arbitrarily deep architectures.
Representation Learning: Mathematical proof of how interactions at finite width enable hidden layer representations to adapt to data.

Official Book Website Free arXiv Preprint (2106.10165) Cambridge University Press Goodfellow et al. Deep Learning Book Understanding Deep Learning (Prince)