Help or Hinder? Evaluating Fairness Metrics and Algorithms in Visualization Systems for Consensus Ranking

Shrestha, Hilson

Etd

Help or Hinder? Evaluating Fairness Metrics and Algorithms in Visualization Systems for Consensus Ranking

Public Deposited

For applications where multiple stakeholders provide recommendations, a fair consensus ranking must not only ensure that the preferences of rankers are well represented, but must also mitigate disadvantages among socio-demographic groups in the final result. However, there is little empirical guidance on the value or challenges of visualizing and integrating fairness metrics and algorithms into human-in-the-loop systems to aid decision-makers. In this work, we design and develop a system called FairFuse, that includes visual encodings of fairness metrics and fair-rank generation algorithms to generate fair consensus rankings. We design a study to analyze the effectiveness of integrating such fairness metrics-based visualization and algorithms. We explore performance through a task-based crowdsourced experiment comparing FairFuse with a similar system for constructing a fair consensus rankings without the inclusion of fairness metrics visualization and algorithms, called ConsensusFuse. We analyze metrics of fairness, agreement of rankers’ decisions, and user interactions in constructing the fair consensus ranking across these two systems. In our study with 200 participants, results suggest that providing fairness-oriented support features nudges users to align their decision with the fairness metrics while minimizing the tedious process of manually having to amend the consensus ranking. We discuss the implications of these results for the design of next-generation fairness oriented- systems, along with emerging directions for future research.

Creator