In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation

Evaluation paths describing decisions made in 86 studies evaluating uncertainty visualizations

About

This page contains information associated with the IEEE VIS 2018 publication:

Hullman, Jessica; Qiao, Xiaoli; Correll, Michael; Kale, Alex & Kay, Matthew. In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation. IEEE Transactions on Visualization & Computer Graphics, 2019 Download PDF

Abstract

Understanding the uncertainty present in data is critical to effectively reasoning about visualized data. However, evaluating the impact of an uncertainty visualization is complex due to the difficulties that people have interpreting uncertainty and the challenge of defining correct behavior with uncertainty information. Currently, evaluators of uncertainty visualization must rely on general purpose visualization evaluation frameworks which can be ill-equipped to provide guidance with the unique difficulties of assessing response to uncertainty. To help evaluators navigate these complexities, we present a taxonomy for characterizing decisions related to evaluating the impact of uncertainty visualizations. Our taxonomy differentiates six levels of decisions that comprise an uncertainty visualization evaluation: the research questions, expected effects from an uncertainty visualization, evaluation goals, measures, elicitation techniques, and analysis paradigms. Applying our taxonomy to 86 user studies of uncertainty visualizations, we find that existing evaluation practice, particularly in visualization research, focus on Performance and Satisfaction-based measures that assume more predictable and statistically-driven judgment behavior than is suggested by research on human judgment and decision making. We reflect on common themes in evaluation practice concerning the interpretation and semantics of uncertainty, the use of confidence reporting, and the disparity between evaluating decision making versus accuracy. We conclude with a concrete set of recommendations for evaluators designed to reduce the mismatch between the conceptualization of uncertainty in visualization versus other fields.

More diagrams

Data Reference

List of all the publications