AlegsaOnline.com

Statistical errors and residuals: definitions, properties and applications

An accessible overview of statistical errors and residuals: what they are, how they differ, common types and properties, example illustrations, and their role in modelling and diagnostics.

Author: Leandro Alegsa Creation date: November 13, 2022 Last updated: May 6, 2026

Measurements are never perfectly exact, and the gap between what we observe and what is true is central to statistics. Practical measurement produces data that contain variability for many reasons: natural variation, limitations of instruments, and sampling procedures. Understanding that variability—expressed as errors and residuals—lets analysts estimate uncertainty, check models, and make informed decisions.

Core definitions and the essential distinction

In statistical usage the term "error" commonly denotes the unobservable difference between an individual observation and the unknown true quantity or value implied by a population model. A "residual" is the corresponding observable quantity computed from a fitted model or a sample: it is the difference between the observed value and an estimated or predicted value. Thus errors are theoretical and generally unobservable; residuals are empirical and used to assess models.

Types and characteristic properties

Sampling error: variation that arises because a sample is only a subset of a population. For a sample mean the deviation from the population mean is a sampling error.
Measurement error: inaccuracies introduced by instruments or protocols during measurement.
Model error: discrepancy between a model's predictions and the true data generating process; this is often the target when estimating parameters.
Systematic vs random: systematic errors bias results in one direction; random errors have no fixed sign and can cancel on average.

When a random process or random variable is involved, the mean or central tendency of the population (population mean) is a theoretical benchmark; the sample mean (sample mean) is the estimator. The difference between an individual observation and the population mean is an error; the difference between the observation and the sample mean is a residual.

Simple example

Imagine an experiment measuring adult heights drawn from a local population. If the true population average height is unknown (but the model posits a central value under the assumed distribution), any single person's measured height minus that true average is a statistical error. If we instead compute the average from a sample of people and subtract that sample mean from each measured height, those differences are residuals. Residuals within a simple random sample sum to zero by construction, whereas true errors need not sum to zero because the population mean is not used to center them. This contrast explains why residuals can display dependencies introduced by estimation even when underlying errors are independent.

Why the distinction matters in practice

Residuals are the practical tool for diagnostics: plotting residuals versus fitted values, checking for patterns, and testing for unequal variance (heteroskedasticity) or serial correlation. Analysts compute residual-based measures—such as residual standard error or sums of squared residuals—to compare models and quantify fit. Because residuals are computed from estimated parameters, they inherit constraints (for example, in ordinary least squares the residuals sum to zero) which affect their statistical properties and the interpretation of tests.

Theoretical context and historical notes

Estimators and their residuals are central to regression methods that date back to the early development of least squares. Under assumptions formalized in results such as the Gauss–Markov theorem, certain estimators (and thus their induced residuals) enjoy optimality properties: they are best linear unbiased estimators of coefficients and their residuals carry information about remaining unexplained variation. Because errors are conceptual and residuals observable, many inferential procedures focus on residuals to assess whether modelling assumptions appear reasonable given the collected data.

Practical reminders and common pitfalls

Do not treat residuals as if they were independent draws of the true errors; estimation constraints can induce dependence.
Differentiate between bias (systematic error) and random variability; both affect conclusions but require different remedies.
Use graphical checks and formal tests on residuals before trusting standard errors and confidence intervals.

In summary, the statistical "error" is the theoretical departure from an unknown truth, while the "residual" is what we compute from observed values and fitted models to assess that departure. Knowing the distinction helps analysts design better studies, choose appropriate estimators, and use residual diagnostics to improve models. For further reading on these concepts consult introductory texts on estimation and regression or follow specific practice-oriented guides linked here: difference, observed value, population.

Disturbance and residual

Disturbance variables are not to be confused with residuals. One distinguishes the two concepts as follows:

Unobservable random perturbations ε $\varepsilon_i$ : measure the vertical distance between observation point and theoretical (true straight line).
Residual ε ${\hat {\varepsilon }}_{i}=y_{i}-{\hat {y}}_{i}$ : Measure the vertical distance between empirical observation and the estimated regression line.

Simple linear regression

→ Main article: Linear simple regression

In the simple linear regression with the single linear regression model the ordinary residuals $Y_{i}=\beta _{0}+\beta _{1}x_{i}+\varepsilon _{i}$ are given by

${\hat {\varepsilon }}_{i}=y_{i}-{\hat {y}}_{i}=y_{i}-{\hat {\beta }}_{0}-{\hat {\beta }}_{1}x_{i}$ .

These are residuals, since an estimated value is subtracted from the true value. More precisely, the fitted values are subtracted from the observed values y_{i}} $y_{i}$ ${\hat {y}}_{i}={\hat {\beta }}_{0}+{\hat {\beta }}_{1}x_{i}$ . In simple linear regression, numerous assumptions are usually made on the confounding variables (see Assumptions about confounding variables).

Residual variance

(also called residual variance) is an estimate of the variance of the regression function in the population $\operatorname {Var} (y\mid X=x)=\operatorname {Var} (\beta _{0}+\beta _{1}x+\varepsilon )=\sigma ^{2}=\operatorname {konst}$ . In simple linear regression, an estimate found by maximum likelihood estimation is given by.

${\tilde {s}}_{\varepsilon }^{2}={\frac {1}{n}}\sum \limits _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}={\frac {1}{n}}\sum \limits _{i=1}^{n}(y_{i}-{\hat {\beta }}_{0}-{\hat {\beta }}_{1}x_{i})^{2}$ .

However, the estimator does not meet common quality criteria for point estimators and is therefore not often used. For example, the estimator is not expectation-true for σ $\sigma ^{2}$ . In simple linear regression, under the assumptions of the classical model of linear simple regression, it can be shown that an expectation-true estimate of the variance of the confounding variables σ $\sigma ^{2}$ , i.e. .an estimate that $\operatorname {E} ({\hat {\sigma }}^{2})=\sigma ^{2}$ is satisfied, given by the variant adjusted for the number of degrees of freedom:

${\hat {\sigma }}^{2}={\frac {1}{n-2}}\sum \limits _{i=1}^{n}(y_{i}-{\hat {\beta }}_{0}-{\hat {\beta }}_{1}x_{i})^{2}$ .

The positive square root of this expectation-stratified estimator is also referred to as the standard error of the regression.

Residuals as a function of the disturbance variables

In simple linear regression, the residuals as a function of the confounding variables ε $\varepsilon_i$ for each individual observation can be written as

${\hat {\varepsilon }}_{i}=\varepsilon _{i}-({\hat {\beta }}_{0}-\beta _{0})-({\hat {\beta }}_{1}-\beta _{1})x_{i}$ .

Sum of the residuals

The KQ regression equation is determined so that the residual sum of squares becomes a minimum. Equivalently, this means that positive and negative deviations from the regression line balance each other out. If the model of the linear single regression contains a - non-zero - intercept, then it must therefore hold that the sum of the residuals is zero

$\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}=0$

Questions and Answers

Q: What is meant by statistical errors and residuals?

A: Statistical errors and residuals refer to the difference between the observed or measured value and the real value, which is unknown.

Q: How can one measure accuracy of a measurement?

A: One can measure the same thing again and again, and collect all the data together. This allows us to do statistics on the data in order to determine how accurate a measurement is.

Q: What is an example of a statistical error?

A: An example of a statistical error would be if there was an experiment to measure the height of 21-year-old men from a certain area with an expected mean of 1.75m, but one man chosen at random was 1.80m tall; then the "(statistical) error" would be 0.05m (5cm).

Q: What is an example of a residual?

A: An example of a residual would be if there was an experiment to measure the height of 21-year-old men from a certain area with an expected mean of 1.75m, but one man chosen at random was 1.70m tall; then the residual (or fitting error) would be -0.05m (-5cm).

Q: Are residuals independent variables?

A: No, The sum of the residuals within a random sample must be zero so they are not independent variables.

Q: Are statistical errors independent variables?

A: Yes, The sum of the statistical errors within a random sample need not be zero; therefore they are independent random variables if individuals are chosen from population independently.

Q: Is it possible to do exact measurements?

A:No, it is not possible to do exact measurements because measurement is never exact