Lecture Schedule (updated throughout the term)

Lecture Materials

• [Notation]
• Unit 1: Introductory Material
  • Knowledge Based System(KBS) versus Machine Learning(ML) (Discrete versus Continuous Math)
  • Addition/Multiplication with KBS vs ML
  • Nearest Neighbors
  • Polynomial Interpolation, Overfitting, Underfitting, Regularization
  • Noise versus Features
  • Monomial Basis and Singular Matrices
  • Error Sources
  • Polynomial Interpolation (Lagrange/Newton basis functions)
  • Representation Theory, CNNs for cloth
• Unit 2: Solving Linear Systems
  • Systems of linear equations
  • Normalization
  • Rank and Solvability
  • Square, Diagonal, Upper Triangular, and Lower Triangular Matrices
  • Gaussian Elimination
  • LU Factorization
  • Pivoting
  • Sparsity
• Unit 3: Understanding Matrices
  • Eigenvalues, eigenvectors, and matrix preconditioning
  • Singular Value Decomposition
  • Singular Matrices and Condition Number
  • Vector and Matrix Norms
• Unit 4: Special Matrices
  • Diagonally Dominant Matrices
  • Symmetric Positive Definite (SPD) Matrices
  • Cholesky Factorization
  • Symmetric Approximation
• Unit 5: Iterative Solvers
  • Continuous Collision Detection
  • Residual
  • Line Search, Steepest Descent
  • Conjugate Gradient (CG) Method
• Unit 6: Local Approximations
  • Taylor Series
  • Sampling
  • Well-Resolved Functions
  • Piecewise Approximation: Constant, Linear, Higher Order Interpolation
  • Cubic Splines
  • 2D Image Segmentation
• Unit 7: Curse of Dimensionality
  • Numerical Integration
  • Newton-Cotes Quadrature
  • Gaussian Quadrature
  • Domain Approximation Errors
  • Curse of Dimensionality
  • Monte Carlo Methods
• Unit 8: Least Squares
  • Eliminating Basis Functions
  • Solving Linear Systems
  • Weighted Minimization
  • Least Squares
• Unit 9: Basic Optimization
  • Critical Points
  • Classifying Critical Points
  • Quadratic Form
  • Least Squares
• Unit 10: Solving Least Squares
  • Normal Equations
  • Condition Number
  • Summary
  • Understanding Least Squares
  • Orthogonal Matrices
  • Gram-Schmidt
  • QR Factorization
  • Householder
• Unit 11: Zero Singular Values
  • Underdetermined Systems
  • Minimum Norm Solution
  • Pseudo-Inverse
  • Sum of Rank One Matrices
  • Matrix Approximation
  • Principal Component Analysis (PCA)
  • Rank One Updates
  • Condition Number of Eigenproblems
  • QR Iteration
  • Power Method
• Unit 12: Regularization
  • Adding an Identity Matrix
  • Full Rank Scenario
  • Rank Deficient Scenario
  • Initial Guess
  • Iterative Approach
  • Adding a Diagonal Matrix
  • Column Space Search Method
• Unit 13: Optimization
  • Function Approximation
  • Choice of Norm
• Unit 14: Nonlinear Systems
  • Jacobian Matrix
  • Linearization
  • Iterative Solver
  • Line Search with Search Directions
• Unit 15: Root finding
  • Fixed Point Iteration
  • Convergence Rate
  • Newton's Method, Secant Method, Bisection Method, and Mixed Methods
  • Nonlinear Systems Problems and Optimization
• Unit 16: 1D Optimization
  • Leveraging Root Finding
  • Unimodal
  • Successive Parabolic Interpolation, Golden Section Search, and Mixed Methods
  • Nonlinear Systems Problems and Optimization
• Unit 17: Computing Derivatives
  • Differentiability and Smoothness
  • Activation Functions
  • Heaviside, Sigmoid, Rectifier, Softplus, Leaky Rectifier, Arg/Soft Max
  • Inequality Constrained Optimization
  • Binary Classification and Soft-margin
  • Symbolic Differentiation
  • Finite Differences
  • Automatic Differentiation
  • Dropout
  • Function Layers
• Unit 18: Avoiding Derivatives
• Broyden's Method
• Symmetric Rank 1
• DFP (Davidon-Fletcher-Powell)
• BFGS (Broyden-Fletcher-Goldfarb-Shanno)
• L-BFGS (Limited Memory BFGS)
• Nonlinear Least Squares
• Gauss Newton and Weighted Gauss Newton
• Levenberg-Marquardt (Damped Nonlinear Least Squares)
• Gradient/Steepest Descent
• Coordinate Descent
• Unit 19: Descent Methods
• Gradient/Steepest Descent
• Steepest Descent for Quadratic Form
• Steepest Descent for Nonlinear Least Squares
• Unit 20: Momentum Methods
• Ordinary Differential Equations (ODEs)
• Gradient Flow
• Families of Solutions and Initial Value Problems (IVPs)
• Well Posedness, Stability, Acccuracy
• Forward Euler, Runge-Kutta, TVD-RK
• Stability Analysis
• Adaptive Time Stepping and Adaptive Learning Rates
• Implicit Methods, Backward Euler, Trapezoidal Rule
• Implicit Stochastic Gradient Descent (ISGD)
• Momentum, Newton's Second Law, First Order Systems
• Momentum Method, Nesterov Momentum, ADAM
• Newmark Methods