Paper Authors

Abstract

NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract

Session 1520

Benchmarks - Are they Really Useful? A Boyanich, S P Maj Department of Computer Science Edith Cowan University Western Australia iso9660@yahoo.com

Abstract Benchmarking is an important commercial tool, which can be used for quantifying the performance of computer and network equipment. Furthermore, benchmarks are potentially valuable as part of curriculum studies in computer and network technology. Benchmarks may be of value to support the understanding of different architectural features and their effect on equipment performance. In effect the benchmarking results may provide a tangible metric that can be related directly not only to various architectural features but also the interactions between different levels in the memory hierarchy, hence providing a method of quantifying different performances of differing computer architectures and configurations. In this context a wide range of benchmarks were tested using the criteria of: repeatability, comparability, consistency, use of meaningful units etc. The criteria selected are based on the fundamental principles of measurement science. Our results clearly indicated that different benchmarking suites gave significantly different results for the same equipment. Also each benchmark failed to give consistent results, when compared to other benchmarks, even on identical equipment. Furthermore many of the benchmarks provided performance data in arbitrary units that were difficult to relate to expected changes in performance. In effect every benchmark tested failed to meet the evaluation criteria. The authors offer an alternative benchmarking method that is designed to meet such criteria and experimental work to date indicates some success using this new metric. This paper presents results of this work and gives recommendations regarding the use of benchmarks in computer education courses.

1. Introduction Benchmarking is a term used to describe the process of testing either a PC or a selected PC module (e.g. Hard Disc) and obtaining a metric representing the associated performance so as to be used as a comparison between other similar devices. It is simply a test to compare performance that may be used to aid the selection of equipment. There is currently a wide range of benchmarking programs readily available. Primarily these standards fall into one of three categories - trade magazines, standards organizations such as SPEC and TPC (Ideas International) and finally individuals. There is a wide range of Benchmarks and often a collection of them is used, a possible advantage being that others

Proceedings of the 2001 American Society for Engineering Education Annual Conference & Exposition Copyright  2001, American Society for Engineering Education

• Citation

• Format
  • APA
  • APA - LaTeX bibitem
  • MLA
  • MLA - LaTeX bibitem
  • Bibtex
  • EndNote - RIS

Boyanich, A., & Maj, S. P. (2001, June), Benchmarks Are They Really Useful? Paper presented at 2001 Annual Conference, Albuquerque, New Mexico. 10.18260/1-2--8951