Mathematics of Big Data and Machine Learning
The D4M model for big data: associative arrays, graph analysis, power law and Kronecker graphs, and databases, with notes on machine learning.
Search, social media, ad placement, spam filtering, fraud detection, drug discovery and bioinformatics all look for items of interest in vast quantities of data. This course introduces the Dynamic Distributed Dimensional Data Model, D4M, which joins graph theory, linear algebra and databases to handle such big data problems, treating them as signal processing on databases. It combines sparse linear algebra, associative arrays, distributed arrays and triple-store databases such as Accumulo, and later notes add the mathematics of big data and machine learning.
What You Will Achieve
- Represent and query data with associative arrays
- Analyze unstructured and structured data with linear algebraic graph methods
- Explain power law graphs and generate Kronecker graph data
- Apply D4M with MATLAB or GNU Octave and a triple-store database
Program Overview
Associative Arrays and Group Theory
The data model and its mathematics.
- Introduction to D4M
- Using associative arrays
- Group theory
Analyzing Data
Graph and array methods on real data.
- Entity analysis in unstructured data
- Analysis of structured data
- Perfect power law graphs
- Bio sequence cross correlation
Databases, Performance and AI
Running at scale, and the later material on AI.
- Accumulo and D4M demo
- Kronecker graphs, data generation and performance
- Notes on AI, machine learning and data architecture
Course Completion
Job Outlook
These methods apply to data science, algorithm development, cyber network data processing and large-scale database work.
Good for: Data scientists and algorithm developers who want a linear algebra route to analyzing big data.
Keep in mind: The videos are an eight-lecture course taught at MIT Lincoln Laboratory in fall 2012, with notes added in 2018 and 2020; D4M runs in MATLAB or GNU Octave.
Bottom line: A mathematical take on big data that treats graphs, arrays and databases as one problem.
Frequently Asked Questions
What is D4M?
The Dynamic Distributed Dimensional Data Model, which combines sparse linear algebra, associative arrays, fuzzy algebra, distributed arrays and triple-store databases such as Hadoop HBase and Apache Accumulo into one database and computation system.
Can I run the software?
Yes. D4M can be downloaded from its website with the code examples used in class, and it supports MATLAB, GNU Octave, Accumulo and SQL through JTDS bindings.
Is there a companion book?
Mathematics of Big Data: Spreadsheets, Databases, Matrices, and Graphs by J. Kepner and H. Jananthan is listed as motivational material.



There are no reviews yet.