What is precision in a numeric value?
Q: What is precision in a numeric value?
A: Precision in a numeric value describes the number of digits that are used to show that value.
Q: How can precision be used to describe the position at which an inexact result will be rounded?
A: Precision can be used to describe the position at which an inexact result will be rounded by setting a given or fixed precision, which is the length of the resulting significand. In financial calculations, a number is often rounded to a given number of places (for example, two places after the decimal separator for many world currencies).
Q: How can 12.345 be expressed with various numbers of significant digits or decimal places?
A: 12.345 can be expressed with various numbers of significant digits or decimal places by rounding it to fit the available precision using round-to-even method.
Q: What happens when insufficient precision is available?
A: When insufficient precision is available, then the number is rounded in some manner to fit the available precision.
Q: Is it appropriate to display a figure with more digits than that which can be measured?
A: No, it is not appropriate to display a figure with more digits than that which can be measured as this creates false precision. For instance, if a device measures to the nearest gram and gives a reading of 12.345 kg, it would create false precision if the measurement were expressed "12.34500 kg" with 2 extra zeroes ("00") at the end.
Q: What formula represents positive numbers x to a precision p significant digits?
A: The formula representing positive numbers x to a precision p significant digits has numerical value given by round(10−n·x)·10n where n = floor(log10 x) + 1 − p . For negative numbers,the numerical value is minus that of its absolute value and 0 has any precison taken as 0
