Evaluating Expert Systems in a Financial Domain: A Study Demonstrating Issues in Case Selection,Validation Standards and Evaluator Bias

This article describes and generalizes a validation study of four commercially available personal financial planning expert systems and the rationale for the research methodology used. Our evaluation of these systems adds to the understanding of verification and validation issues related to case selection, validation standards and evaluator bias. The article describes the systems, their domain and the empirical method—field tests using hypothetical cases—and relates that method to the literature. Comparing same-task systems combines multiple system perspectives and multiple models. Our methodology did efficiently and effectively identify conflicting terminology, omissions and system weaknesses but was inadequate for comparing the complex plan recommendations. The results re-emphasize the importance of continual knowledge base updating, formal system testing and the need for external evaluation. The results also show the value of comparing multiple, same-task systems.