New measurements for search engine evaluation proposed and tested

Authors

L. Vaughan, Vaughan, L.; Fac. of Info. and Media Studies, University of Western Ontario, London, Ont. N6A 5B7, Canada; email: lvaughan@uwo.caFollow

Document Type

Article

Publication Date

1-1-2004

Volume

40

Journal

Information Processing and Management

First Page

677

URL with Digital Object Identifier

https://doi.org/10.1016/S0306-4573(03)00043-8

Last Page

691

Abstract

A set of measurements is proposed for evaluating Web search engine performance. Some measurements are adapted from the concepts of recall and precision, which are commonly used in evaluating traditional information retrieval systems. Others are newly developed to evaluate search engine stability, an issue unique to Web information retrieval systems. An experiment was conducted to test these new measurements by applying them to a performance comparison of three commercial search engines: Google, AltaVista, and Teoma. Twenty-four subjects ranked four sets of Web pages and their rankings were used as benchmarks against which to compare search engine performance. Results show that the proposed measurements are able to distinguish search engine performance very well. © 2003 Elsevier Ltd. All rights reserved.