Matrix Calculus for Machine Learning and Beyond
Course facts
- Payment: Free, no certificate
- Credits: Non Credit
- Learning style: Self-paced, start anytime
- Resources: Editable Files; Instructor Insights; Lecture Notes; Lecture Videos; Problem Set Solutions; Problem Sets; Readings
MIT OpenCourseWare details
- As taught in: January IAP 2023
- Department: Mathematics
| Course Details | ||||||
|---|---|---|---|---|---|---|
|
||||||
| Outcomes | ||||||
|
||||||
| Price & Terms | ||||||
|
||||||
| Technical | ||||||
|
||||||
| Specification | ||||||
|
||||||
Derivatives for matrices and vector spaces: matrix factorizations, chain rules, automatic differentiation, adjoint methods, and Hessians.
Machine learning and large-scale optimization need a step beyond single-variable and vector calculus: calculus on matrices and on arbitrary vector spaces. This short course treats a matrix as a whole rather than an array of scalars, defines derivatives as linear operators, and works out derivatives of inverses, determinants and matrix factorizations, chain rules on computational graphs, forward- and reverse-mode differentiation, adjoint methods and second derivatives.
What You Will Achieve
- Treat derivatives as linear operators on arbitrary vector spaces
- Differentiate functions of matrices, including inverses, determinants, and factorizations
- Apply forward- and reverse-mode differentiation and chain rules on computational graphs
- Use adjoint methods, Hessians, and Newton methods in optimization
Program Overview
Derivatives as Linear Operators
A coherent foundation for derivatives beyond gradients and Jacobians.
- Linear approximation on arbitrary vector spaces
- Derivatives with matrix inputs and outputs
- Kronecker products and matrix vectorization
Chain Rules and Automatic Differentiation
Composing derivatives, by hand and by computer.
- Multidimensional chain rules and composition order
- Chain rules on computational graphs such as neural networks
- Forward- and reverse-mode automatic differentiation
- Derivatives of eigenvalues, the SVD, and constrained matrices
Adjoint Methods and Optimization
Derivatives of solutions, and how they drive optimization.
- Adjoint methods for linear, nonlinear, and differential equations
- Nonlinear root-finding and optimization with Newton and steepest descent
- Second derivatives, Hessians, and quasi-Newton methods
- Applications in engineering, science, and machine learning
Course Completion
Job Outlook
Matrix calculus is used in machine learning research and engineering, scientific computing, optimization and any work that trains models or tunes large systems.
Good for: Learners heading into machine learning or optimization who want the derivatives behind them worked out properly.
Keep in mind: It is a short January course with two problem sets, so it introduces each idea rather than drilling it; the full lecture notes are the textbook.
Bottom line: A compact, recorded course that rethinks derivatives for matrices, with complete notes and solved problem sets.
Frequently Asked Questions
Is there a textbook for the course?
No textbook is required. A complete set of lecture notes, prepared by Paige Bright with the two professors, is provided and is also available on arXiv.
Does the course cover automatic differentiation?
Yes. It treats forward- and reverse-mode manual and automatic differentiation, and chain rules on computational graphs such as neural networks.
Are the problem set solutions published?
Yes. Both problem sets are published with their solutions, the second also as a Jupyter notebook.
User Reviews
Write your review
You must be logged in to post a review.
Reviews can only be posted by registered learners. Log in or create a free account, then you will return here to post.
There are no reviews yet.