Linux Benchmarking - Article III - Interpreting benchmark results: An example of correct/incorrect interpretation of results

## 5. An example of correct/incorrect interpretation of results

We finally get to the practical part of this article. As usual, I propose a different benchmark as a practical example, only this time we will be seeing a more complex benchmark, in fact a CPU benchmark suite: we'll use the latest version of nbench-byte (version 2.1) as our example. You can download it from Uwe Mayer's new Web site or from the Linux Benchmarking Project

What we are going to measure this time can be described as "general CPU performance". So: this is not processor performance for matrix operations, this is not MMX performance, this is not the ability of a processor to decode an MPEG stream. Also, this is not a measure of a processor interrupt response time, peak MIPS, etc.

1. Wrong ways to benchmark.
2. Wrong way to analyze benchmarking data measurements.

Now let's take a look at a correct procedure, following all the steps recommended in section 1.1

## 5.1 Example

### Stating our objective

For this short example we just want to compare the performance of two different CPUs: the AMD K6 and the Cyrix 6x86MX. This is comparative benchmarking, so we should keep all conditions fixed and vary just this single variable: the CPU.

This is not too ambitious and I have no bias for/against any of these two chips. Also, since both CPUs are widely available at reasonable prices, such comparative benchmarking may be of interest to GNU/Linux users wanting to upgrade and/or put together their next CPU.

### Choice of a benchmark

Nbench-byte is an improved, updated version of the BYTEmark benchmark suite developed at BYTE magazine by Rick Grehan. Uwe F. Mayer did the port to Linux and is its present maintainer/developer. The latest version is 2.1, dated December 97.

Similarly to SPEC95, this modern CPU benchmark suite uses 10 different algorithms that are representative of common CPU-intensive tasks (the file bdoc.txt included with the source has a description of each algorithm). Note that Rick has stopped development of BYTEmark (neither Uwe nor myself managed to contact him), but you can see that this is not a committee-designed benchmark; in this respect its lineage fits quite well the GNU/Linux style of development.

Nbench-byte 2.1 also goes one step beyond SPEC95 in that it generates three index figures: an Integer Index, a Floating-Point Index and a Memory Index. The Memory Index reflects the fact that on most modern CPUs, the memory subsystem represents a major performance bottleneck. You can check the Web site for STREAM a new benchmark specifically created to address this issue, for more information on this topic.

One of nbench-byte nicest features is that it calibrates itself. For each of the tests it determines a minimum amount of work that needs to be done to be able to accurately measure the time needed. Then it runs that test five times and does a statistical analysis (using the student-t distribution) to see if the results are consistent (meaning that the probability is at least 95% that the true mean of the results is within 5% of the calculated mean of the results). If not, then nbench runs the test up to twenty-five times more and does the statistical analysis after each additional test run. If consistency cannot be achieved within a total of thirty runs, a warning will be issued when the score gets reported.

In terms of raw data statistical processing, nbench-byte 2.1 goes beyond all the other benchmarks I have ever come across.

Another very interesting feature of this benchmark suite is its portability across a wide range of OS's and platforms. However, because of fundamental differences in compiler/libraries/memory management in different OSes, this benchmark should not be carelessly used to compare results across platforms. This is not an OS benchmark, it's a CPU benchmark (see the pitfalls subsection below). You have been warned.

### Benchmark setup

We are doing comparative benchmarking, so we will be using exactly the same hardware for our benchmark runs. All that will change between runs is:

1. The processor (one run with a 6x86MX, the other run with a K6).
2. A small cyrix.rc file that was added to the rc.local script. This calls set6x86 to setup a few internal 6x86MX registers. The K6 does not need this file.

Also note that we are using the precompiled nbench executable, as shipped in the tar.gz package.

To describe our hardware setup, we resort to the Linux Benchmarking Toolkit Report Form:

LINUX BENCHMARKING TOOLKIT REPORT FORM

    CPU

    ===

    Vendor: AMD/Cyrix

    Model: K6-166/6x86MX-PR200

    Core clock:166 MHz (2.5 x 66MHz)

    Motherboard vendor: ASUS

    Mbd. model: P55T2P4

    Mbd. chipset: Intel HX

    Bus type: PCI

    Bus clock: 33 MHz

    Cache total: 512 Kb

    Cache type/speed: Pipeline burst 6 ns

    SMP (number of processors): 1

    RAM

    ===

    Total: 32 MB

    Type: EDO SIMMs

    Speed: 60 ns

    Disk

    ====

    Vendor: IBM

    Model: IBM-DCAA-34430

    Size: 4.3 GB

    Interface: EIDE

    Driver/Settings: Bus Master DMA mode 2

    Video board

    ===========

    Vendor: Generic S3

    Model: Trio64-V2

    Bus: PCI

    Video RAM type: 60 ns EDO DRAM

    Video RAM total: 2 MB

    X server vendor: XFree86

    X server version: 3.3

    X server chipset choice: S3 accelerated

    Resolution/vert. refresh rate: 1152x864 @ 70 Hz

    Color depth: 16 bits

    Kernel

    ======

    Version: 2.0.29

    Swap size: 64 MB

    gcc

    ===

    Version: 2.7.2.3

    Options: (default nbench)

    libc version: 5.4.38

    Test notes

    ==========

    Two processors tested. The 6x86MX was configured with a special rc.cyrix file.

    RESULTS

    ========

    Linux kernel 2.0.0 Compilation Time: N/A

    Whetstone Double Precision (FPU) INDEX: N/A

    UnixBench 4.10 system INDEX: N/A

    Xengine: N/A

    nbench-byte integer INDEX: 6x86MX - 0.686; K6 - 0.713

    nbench-byte memory INDEX: 6x86MX - 0.753; K6 - 0.793

    nbench-byte floating-point INDEX: 6x86MX - 0.655; K6 - 0.802

    Comments

    =========

    With the CPU case open, it took me 30 minutes to run nbench-byte on the two processors!

### Detailed benchmark results

One can get very detailed benchmark results with nbench-byte 2.1 by specifying the -v option. However, here we are only showing the normal output from a standard run, first on the 6x86MX, then on the K6:

6x86MX results:

``

`BYTEmark* Native Mode Benchmark ver. 2 (10/95)`

`Index-split by Andrew D. Balsa (11/97)`

`Linux/Unix* port by Uwe F. Mayer (12/96,11/97)`

``

`TEST : Iterations/sec. : Old Index : New Index`

`: : Pentium 90* : AMD K6/233*`

`--------------------:------------------:-------------:------------`

`NUMERIC SORT : 80.681 : 2.07 : 0.68`

`STRING SORT : 11.107 : 4.96 : 0.77`

`BITFIELD : 2.1997e+07 : 3.77 : 0.79`

`FP EMULATION : 8.5349 : 4.10 : 0.95`

`FOURIER : 881.21 : 1.00 : 0.56`

`ASSIGNMENT : 0.71582 : 2.72 : 0.71`

`IDEA : 147.28 : 2.25 : 0.67`

`HUFFMAN : 58.095 : 1.61 : 0.51`

`NEURAL NET : 0.70897 : 1.14 : 0.48`

`LU DECOMPOSITION : 27.869 : 1.44 : 1.04`

`==========================ORIGINAL BYTEMARK RESULTS==========================`

`INTEGER INDEX : 2.861`

`FLOATING-POINT INDEX: 1.181`

`Baseline (MSDOS*) : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0`

`==============================LINUX DATA BELOW===============================`

`C compiler : gcc version 2.7.2.3`

`libc : libc.so.5.4.38`

`MEMORY INDEX : 0.753`

`INTEGER INDEX : 0.686`

`FLOATING-POINT INDEX: 0.655`

`Baseline (LINUX) : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38`

`* Trademarks are property of their respective holder.`

K6 results:

``

`BYTEmark* Native Mode Benchmark ver. 2 (10/95)`

`Index-split by Andrew D. Balsa (11/97)`

`Linux/Unix* port by Uwe F. Mayer (12/96,11/97)`

``

`TEST : Iterations/sec. : Old Index : New Index`

`: : Pentium 90* : AMD K6/233*`

`--------------------:------------------:-------------:------------`

`NUMERIC SORT : 82.229 : 2.11 : 0.69`

`STRING SORT : 10.57 : 4.72 : 0.73`

`BITFIELD : 2.0672e+07 : 3.55 : 0.74`

`FP EMULATION : 6.4842 : 3.11 : 0.72`

`FOURIER : 1117.1 : 1.27 : 0.71`

`ASSIGNMENT : 0.93388 : 3.55 : 0.92`

`IDEA : 158.42 : 2.42 : 0.72`

`HUFFMAN : 81.407 : 2.26 : 0.72`

`NEURAL NET : 1.0764 : 1.73 : 0.73`

`LU DECOMPOSITION : 26.521 : 1.37 : 0.99`

`==========================ORIGINAL BYTEMARK RESULTS==========================`

`INTEGER INDEX : 2.990`

`FLOATING-POINT INDEX: 1.445`

`Baseline (MSDOS*) : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0`

`==============================LINUX DATA BELOW===============================`

`C compiler : gcc version 2.7.2.3`

`libc : libc.so.5.4.38`

`MEMORY INDEX : 0.793`

`INTEGER INDEX : 0.713`

`FLOATING-POINT INDEX: 0.802`

`Baseline (LINUX) : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38`

`* Trademarks are property of their respective holder.`

### Data analysis

We will concentrate on the Linux data, for obvious reasons. As we can see, whereas the 6x86MX outperforms the K6 on some tests by a narrow margin (approx. 6%), the K6 vastly outperforms the 6x86MX on other tests.

### Conclusion

On our synthetic test nbench-byte version 2.1, the K6 has shown slightly better performance than the 6x86MX, running at the same 166MHz (2.5 x 66MHz) clock rate on exactly the same hardware.

## 5.2 Pitfalls

The basic pitfall that one should be warned against concerning nbench-byte applies similarly to all benchmarks: one should not to try to use this tool for something it was not designed for. Since this is a CPU benchmark, do not use it to test OS performance, video bandwidth, or any other feature that implies I/O activity. Also, it is not an adequate tool for comparing compilers and/or C and math libraries.

This is less obvious than it seems at first. For an accurate, thorough, documented discussion of this particular pitfall, you are referred to one of Uwe's excellent pages on benchmarking

Another pitfall would have been to compare the two processors running on widely different machines. Motherboard, cache and RAM timing setup can skew results by as much as 10%. Compilation options and libraries can also skew results by 25% or more.