Overview
A Markov chain is a mathematical model for a system that moves between a set of discrete states in a sequence of steps, where the probability of the next state depends only on the current one. This condition is called the Markov property. Markov chains formalize the idea of a memoryless stochastic process and are used to describe phenomena in physics, biology, computer science, economics and many other fields. For a concise statement of the property and basic definitions see Markov property reference.
Core concepts and components
A typical Markov chain is specified by:
- State space — the set of possible states the process can occupy (finite or countable).
- Initial distribution — the probability of starting in each state.
- Transition mechanism — rules that determine the probability of moving from one state to another. In discrete time this is given by a transition matrix; in continuous time it is encoded by transition rates.
When time is considered in discrete steps the model is called a Discrete Time Markov Chain (DTMC); for models where transitions can occur at any real-valued time and holding times follow exponential laws, one has Continuous Time Markov Chains (CTMC). See a short comparison at discrete and continuous chains.
Mathematical formulation and properties
For a chain with a finite or countable state space, the one-step transition probabilities p(i,j) give P(X_{t+1}=j | X_t=i). These probabilities are arranged in a transition matrix P for discrete chains. Key analytic concepts include:
- Stationary distribution — a probability vector π satisfying πP = π; if it exists and the chain is ergodic, long-run state frequencies converge to π.
- Irreducibility and recurrence — classifications that describe whether states communicate and whether they are revisited infinitely often.
- Absorbing states — states that, once entered, cannot be left; these influence eventual behaviour and absorption probabilities.
Continuous-time chains are often specified by a generator matrix of rates; the time spent in a state is commonly modeled by an exponential distribution, whose memoryless property aligns naturally with Markov assumptions. For a primer on transition probabilities and exponential holding times, consult transition probabilities and exponential holding times.
Long-run behaviour and computations
Analysts study Markov chains to predict steady-state behaviour, mixing times (how long until the chain is close to its stationary law), and absorption probabilities. For finite ergodic chains, repeated multiplication by the transition matrix often reveals convergence to a unique stationary vector. For applied problems one computes powers of P, solves linear systems for π, or simulates sample paths to estimate expectations and hitting times.
Examples and applications
Simple illustrative examples include weather models that move between states like "sunny" and "rainy", random walks on graphs, and small decision processes. A pedagogical toy model describes an animal that eats one of three foods each day and chooses according to probabilities that depend only on yesterday's meal; this is a DTMC because the choice depends solely on the previous state.
A more applied historical example is a RAND Corporation model of drug-use categories framed as transitions among non-user, light user and heavy user states; such models use systems of difference equations to project population fractions from one time step to the next. See a compact discussion of that formulation at RAND model summary.
History, variations and notable facts
The theory dates to the early 20th century with work by Andrey Markov on chains of dependent events. Since then the framework has expanded to cover hidden Markov models (where states are not directly observed), continuous-state analogues, and Markov decision processes, which introduce control actions. Markov chains underpin algorithms such as Markov Chain Monte Carlo (MCMC) used for sampling complex distributions.
Because of their conceptual simplicity and wide applicability, Markov chains are a foundational tool in probabilistic modelling and computational statistics. Introductory expositions and computational examples can be found through many textbooks and online references; a useful starter link is basic theory, while applied tutorials and simulation guides often point to resources at applied introductions and numerical methods.
Further reading and software-oriented notes are available at other curated resources and educational sites; for general survey material try the links above and specialized discussions of continuous-time formulations at CTMC overview.