Information theory: fundamentals, measures, history and applications

Key concepts and common measures

Several related quantities are central to information theory. They formalize uncertainty, dependence and transmission limits:

• Entropy: a numerical measure of uncertainty or average surprise for the outcomes of a random process; see entropy. It is usually expressed in bits when logarithms base two are used.
• Mutual information: the reduction in uncertainty about one variable given knowledge of another; formally linked to mutual information.
• Relative entropy (Kullback–Leibler divergence): a measure of how one probability distribution diverges from another and a tool for comparing models.
• Channel capacity: the maximum reliable transmission rate of a communication channel, determined by noise and encoding methods.
• Error exponents and reliability functions: quantify how quickly the probability of decoding error can fall as message length increases.

Formal ideas: sources, variables and processes

Information theory models data generators as probabilistic objects: discrete or continuous random variables and random processes. Compression and transmission are statistical in nature — they exploit predictable structure in these probabilistic models. When a source is highly predictable, entropy is low and the data can be compactly represented; when outcomes are nearly equiprobable, entropy is high and more bits are required.

Historical origin and foundational results

The modern subject took shape with Claude Shannon's work in the mid‑20th century, which established the fundamental limits of data compression and communication. Shannon’s theorems show that under suitable probabilistic models there exist codes that approach the theoretical minima for rate and error, and they identify the physical and mathematical bounds that cannot be surpassed by any coding scheme.

Applications and examples

Information theory underpins many technologies and disciplines. Practical areas include lossless and lossy data compression, digital data communication systems, error‑correcting codes, and cryptography. It also informs modern work in machine learning, statistical inference, neuroscience and biology, where the notion of information helps compare models and understand limits of sensing and learning. Because many systems are modeled probabilistically, these applications are inherently statistical in character.

Distinctions and noteworthy facts

Information‑theoretic entropy is conceptually distinct from thermodynamic entropy, although they share mathematical similarities. Units of information (bits, nats) make quantitative trade‑offs explicit: for example, channel capacity tells how many bits per second can be transmitted reliably. Understanding mutual information helps designers balance data rate, latency and error tolerance. For further technical reading consult introductory and advanced treatments that cover coding theorems, algorithmic methods and contemporary extensions to networks and quantum systems.

For more focused material on specific topics linked above, see entries on applied mathematics, electrical engineering, and linked concepts such as information, limits, data compression, data communication, statistical methods, entropy, random variables, random processes and mutual information.