The Statistical Engineering and Mathematical and Computational Sciences Divisions of the Information Technology Laboratory (National Institute of Standards and Technology) has released a number of benchmark datasets for assessing the numerical accuracy of statistical software. The Statistical Reference Datasets (StRD) were designed explicitly to assist researchers in benchmarking statistical software packages. A link to the StRD is maintain on the NIST’s website for applied mathematics, statistics, and computational science{1}.

The Statistical Reference Datasets include both generated and "real-world" data. The StRD{2} are divided into four suites of numerical benchmarks for statistical software:

• univariate summary statistics.
• one-way analysis of variance.
• linear regression.
• nonlinear regression.

For each suite of tests there are several problems in each of three difficulty levels: low, average, and high. These benchmarks were specifically designed so that reliable algorithms implemented in double precision would produce acceptable results for all four suites.

In the case of linear procedures the certified values were calculated using multiple precision calculations (accurate to 500 digits). Data were read in exactly as multiple precision numbers and all calculations were made with this very high precision. For the nonlinear least-squares regression benchmarks, two different algorithms with different implementations were used with 128-bit precision to calculate the certified values. Both of these algorithms can, in double precision (64-bit), return 10 accurate digits for each of the nonlinear benchmarks. In addition, multiple profiles of the least-squares surface were used to ensure that a global minimum had been attained.

The certified values for the benchmarks are provided to 15 digits for linear procedures and 11 digits for nonlinear procedures.

This section gives a brief overview of topics related to software testing. Specifically discussed are the four types of errors associated with numerical computations, the metric to be used for measuring accuracy, and what is an acceptable level of accuracy for engineering assessments

As described in Reference [2], there are four types of errors in numerical computations.

1. A computer represents numbers in binary form using finite precision. For example, by the IEEE-754 standard for computer arithmetic (which is implemented in the hardware of most every computer) single precision has about six or seven digits of accuracy, while double precision has about 15 or 16 digits of accuracy. Often, a given floating-point number cannot be represented exactly in binary form. For example, the binary representation of the decimal number 0.1 is .001100110011, where the underline indicates an infinitely repeating sequence.
2. Rounding error is a function of hardware, and is primarily due to finite precision. One type of rounding error is cancellation error, which occurs when two nearly equal numbers are subtracted from each other, leaving only the rightmost digits. Successive rounding errors in a series of calculations do not cancel but, rather, accumulate, and the bound on the total error is proportional to the number of calculations. Sometimes this total error only affects the rightmost digits of the final answer. Sometimes the total error can be so large as to completely swamp the answer, resulting in no accurate digits.
3. Truncation error is a function of software and may be considered an approximation error. Many mathematical functions can be represented as an infinite series expansion. Assuming infinite precision, the difference between the true value and that achieved by summing a finite number of terms is truncation error.
4. Use of an "unstable" algorithm is a function of software. In general a number of different algorithms can be used to compute a given mathematical quantity. Each has its own strengths and weaknesses, and the algorithm is often chosen based on considerations of speed or accuracy.

Reference [2] uses a measure of the number of correct significant digits in the estimated value as a performance index. This measure of accuracy is an absolute measure of accuracy of an estimate and is easier to apply than the relative performance index (measure) proposed in Reference [5]. The relative performance index of Reference [5] is a measure of the number of figures of accuracy lost by the test software over and above what software based on an optimally stable algorithm would produce. The simpler log relative error (LRE) is used as the basis for performance index in this technical note. Figure 1 is a flowchart for determining the LRE. The base-10 logarithm of the relative error is defined as:

where: LRE is the log relative error. q is the estimated value. c is the correct (certified) value. The LRE is a measure of the number of correct significant digits only when the estimated value is "close" to the exact value. Therefore, each estimated quantity must be compared to its certified value to make sure that they differ by a factor of less than two; otherwise the LRE for the estimated quantity is zero. In the event that the estimated value equals the correct value, the LRE is undefined, in which case it should be set equal to the number of digits in the certified value. In those cases that the correct value is zero, the base-10 logarithm of the absolute error should be used:

The conditioning of the data consists of subtracting the sample mean from all of the values in the sample. In the case of the univariate summary statistics and the analysis of variance procedures, the results for the raw and conditioned benchmarks are mathematically identical, but use of the conditioned data results in a problem that is numerically better conditioned. In the case of the linear regression procedures, the results of the raw and conditioned benchmarks are not mathematically identical. In this case it is necessary to evaluate the regression parameters (in terms of the sample mean and the conditioned regression parameters) for the conditioned data benchmarks so that they can be compared with the raw data benchmarks and the certified values for the dataset.

As described in Reference [2], the LRE from low-difficulty linear procedures should be at least nine. This is because a decent implementation of a decent algorithm for a linear procedure should, with well-behaved data, return ten accurate digits in double precision. This is not to suggest that the user need ten digits of accuracy. Rather, the idea is that a program that cannot deliver ten digits of accuracy for an easy benchmark is more likely to deliver inaccurate results in practice.

Producing correct results for all of the benchmark problems does not imply that the software will do the same for actual datasets. It does, however, provide some degree of assurance that the software does provide correct results for benchmark problems known to yield incorrect results for some software programs.

In practice, the author considers that four digits of accuracy are sufficient for a majority of engineering assessments. The important issue is that the analyst must know of, understand, and appreciate the implications of any/all limitations in the software packages that they use.

[2] McCullough,B.D.

.......

[5] Cook,H.R., Cox,M.G., Dainton,M.P., Harris,P.M.

.......