Asee peer logo

Benchmarks Are They Really Useful?

Download Paper |


2001 Annual Conference


Albuquerque, New Mexico

Publication Date

June 24, 2001

Start Date

June 24, 2001

End Date

June 27, 2001



Page Count


Page Numbers

6.234.1 - 6.234.10



Permanent URL

Download Count


Request a correction

Paper Authors

author page

A Boyanich

author page

S P Maj

Download Paper |

NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract

Session 1520

Benchmarks - Are they Really Useful? A Boyanich, S P Maj Department of Computer Science Edith Cowan University Western Australia

Abstract Benchmarking is an important commercial tool, which can be used for quantifying the performance of computer and network equipment. Furthermore, benchmarks are potentially valuable as part of curriculum studies in computer and network technology. Benchmarks may be of value to support the understanding of different architectural features and their effect on equipment performance. In effect the benchmarking results may provide a tangible metric that can be related directly not only to various architectural features but also the interactions between different levels in the memory hierarchy, hence providing a method of quantifying different performances of differing computer architectures and configurations. In this context a wide range of benchmarks were tested using the criteria of: repeatability, comparability, consistency, use of meaningful units etc. The criteria selected are based on the fundamental principles of measurement science. Our results clearly indicated that different benchmarking suites gave significantly different results for the same equipment. Also each benchmark failed to give consistent results, when compared to other benchmarks, even on identical equipment. Furthermore many of the benchmarks provided performance data in arbitrary units that were difficult to relate to expected changes in performance. In effect every benchmark tested failed to meet the evaluation criteria. The authors offer an alternative benchmarking method that is designed to meet such criteria and experimental work to date indicates some success using this new metric. This paper presents results of this work and gives recommendations regarding the use of benchmarks in computer education courses.

1. Introduction Benchmarking is a term used to describe the process of testing either a PC or a selected PC module (e.g. Hard Disc) and obtaining a metric representing the associated performance so as to be used as a comparison between other similar devices. It is simply a test to compare performance that may be used to aid the selection of equipment. There is currently a wide range of benchmarking programs readily available. Primarily these standards fall into one of three categories - trade magazines, standards organizations such as SPEC and TPC (Ideas International) and finally individuals. There is a wide range of Benchmarks and often a collection of them is used, a possible advantage being that others

Proceedings of the 2001 American Society for Engineering Education Annual Conference & Exposition Copyright  2001, American Society for Engineering Education

Boyanich, A., & Maj, S. P. (2001, June), Benchmarks Are They Really Useful? Paper presented at 2001 Annual Conference, Albuquerque, New Mexico. 10.18260/1-2--8951

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2001 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015