Information and Entropy
Information and entropy: bits, codes, compression, noise, probability, communications, inference, maximum entropy and quantum information.
What are the ultimate limits to communication and computation, and what does information have to do with physics? This course for first-year students explores the physical nature of information and its processing, unit by unit: bits and codes, compression, noise and error correction, probability, communications, processes and inference, then maximum entropy, physical systems, energy, temperature and quantum information. It is taught from a full set of course notes by Professor Penfield.
What You Will Achieve
- Represent information in bits and design codes to store and send it
- Apply compression and error-correcting codes to data in the presence of noise
- Use probability and inference to reason about uncertain information
- Connect the information-theoretic idea of entropy to energy and temperature in physical systems
Program Overview
Bits, Codes and Compression
Representing and packing information.
- Bits and codes
- Compression
- Noise and errors
Probability, Communication and Inference
Information under uncertainty.
- Probability
- Communications
- Processes
- Inference
Entropy and Physics
Where information meets thermodynamics and quantum mechanics.
- Maximum entropy
- Physical systems, energy and temperature
- Quantum information
Course Completion
Job Outlook
Information theory underpins communications engineering, data compression, computer science, statistical physics and quantum computing.
Good for: Curious beginners in science or engineering who want to see how information, entropy and physics fit together.
Keep in mind: The problem sets on the site are representative ones from 2003, some with solutions, and there are no lecture videos; the course notes carry the material.
Bottom line: An unusual, illuminating first look at information as a physical quantity, built on excellent course notes.
Frequently Asked Questions
What are the course notes?
A complete set of notes written by Professor Penfield over several years of teaching the subject, linked unit by unit and published as an open textbook.
Are there problem sets to practice with?
Yes. Problem sets from 2003 are republished with each unit as representative homework, some of them with solutions, along with programming assignments.
How does the course link information to physics?
After coding, compression and inference, the later units apply the principle of maximum entropy to physical systems, energy and temperature, and end with quantum information.



There are no reviews yet.