Overview

Numerical precision expresses how finely a quantity is specified by its digits or by the numeric format used to represent it. Commonly this is reported as the number of decimal places (digits to the right of the decimal point) or as the count of significant figures (the number of meaningful digits in a value). Precision indicates the resolution of a numeric statement: greater precision allows smaller changes in value to be distinguished.

Ways to express precision

  • Decimal places: the count of digits to the right of the decimal point (for example, 54.6 has one decimal place).
  • Significant figures: the number of meaningful digits irrespective of decimal point position (for example, 0.0045 has two significant figures).
  • Implicit integer precision: integers with trailing zeros are ambiguous: 500 is sometimes taken to have two trailing zeros of scale, occasionally described as having precision −2 in a scale-of-ten sense, while 5.00×10^2 or 500. makes greater precision explicit.

Rules and common conventions

General conventions used to interpret digits include: leading zeros (for example, in 0.012) are not significant; trailing zeros after a decimal point are significant (5.00 has three significant figures); trailing zeros in a whole number without a decimal point are ambiguous unless a convention or notation clarifies intent. To avoid ambiguity, scientific notation or an explicit decimal point is often used.

Rounding and uncertainty

Rounding reduces a value to a stated precision and can affect subsequent calculations. Several rounding methods exist: round-half-up (common in basic practice), round-half-to-even (often used in computing for statistical neutrality), and others. In measurement science, precision is closely tied to uncertainty: a reported result is usually accompanied by an uncertainty estimate (for example, 12.3 ± 0.1) or expressed with an implied uncertainty conveyed by the number of significant figures.

Precision versus accuracy

Precision is distinct from accuracy. Precision describes the granularity or repeatability of values, while accuracy describes closeness to a true or accepted value. Measurements or calculations can be precise but inaccurate (systematic bias), or accurate but reported with low precision. Both concepts are important when assessing the reliability of numerical information.

Digital representation and computation

In computing, precision refers to how many digits a numeric data type can represent without loss. Floating-point formats are commonly described as single or double precision to indicate their capacity for storing significant digits and for supporting a numeric range. Fixed-point formats use an implicit number of fractional digits. Finite precision in digital systems leads to round-off error, cancellation (loss of significant digits when subtracting similar numbers), and potential overflow or underflow; numerical analysis uses specialized methods to mitigate these effects.

Practical guidance

  • When reporting measured values, always state units and, if possible, an uncertainty or the number of significant figures intended.
  • Use scientific notation to make magnitude and precision explicit (for example, 5.00×10^2 to show three significant figures for five hundred).
  • Be explicit about rounding rules used when publishing computed results or when combining values of differing precision.

Examples

Simple examples illustrate common usage: 54.6 has one decimal place and three significant figures; 0.0045 has two significant figures; 500 written without further notation can be taken as having two trailing zeros of scale (sometimes described as precision −2) but may be clarified as 5.00×10^2 if additional precision is intended; 5.00 indicates a higher precision than 5.

For further discussion of notation, conventions and formal definitions see additional resources.