Abstract
Enabling non-discrimination for end-users of recommender systems by introducing consumer fairness is a key problem, widely studied in both academia and industry. Current research has led to a variety of notions, metrics, and unfairness mitigation procedures. The evaluation of each procedure has been heterogeneous and limited to a mere comparison with models not accounting for fairness. It is hence hard to contextualize the impact of each mitigation procedure w.r.t. the others. In this paper, we conduct a systematic analysis of mitigation procedures against consumer unfairness in rating prediction and top-n recommendation tasks. To this end, we collected 15 procedures proposed in recent top-tier conferences and journals. Only 8 of them could be reproduced. Under a common evaluation protocol, based on two public data sets, we then studied the extent to which recommendation utility and consumer fairness are impacted by these procedures, the interplay between two primary fairness notions based on equity and independence, and the demographic groups harmed by the disparate impact. Our study finally highlights open challenges and future directions in this field.
Motivation
Recommender systems help us make decisions in various domains, from selecting books to choosing friends. However, their wide adoption has spurred investigations into possibly unfair practices. Group consumer fairness should account for no disparate impact of recommendations on protected groups of consumers, but current research has led to heterogeneous evaluation protocols that make it hard to compare different mitigation approaches.
Research Questions
- RQ1: Is recommendation utility affected by the mitigation procedures?
- RQ2: Do the selected mitigation procedures reduce the unfairness estimates?
- RQ3: Is disparate impact systematically harming the minority group?
Methodology
We conducted a systematic study on algorithmic procedures for mitigating consumer unfairness. We scanned proceedings of top-tier conferences and journals (ACM, Elsevier, IEEE, Springer), identifying 15 relevant papers. Only 8 of them could be reproduced with working source code.
Experimental Setup
We defined a common evaluation protocol using:
- Datasets: MovieLens 1M and LastFM 1K
- Sensitive attributes: Gender and Age
- Fairness notions: Equity (NDCG Demographic Parity) and Independence (Kolmogorov-Smirnov test)
Main Findings
- Utility impact is often negligible (RQ1): Mitigation procedures did not consistently reduce the utility of recommendations
- Unfairness reduction is inconsistent (RQ2): Only a minor subset of procedures substantially reduce unfairness, and rarely for both equity and independence notions simultaneously
- Disparate impact doesn't always harm minorities (RQ3): The minority group was actually advantaged in some settings (e.g., LFM 1K for both attributes in top-n recommendation)
- Reproducibility is challenging: Only 8 out of 15 papers could be reproduced, highlighting the need for better code sharing practices
Open Challenges
- Reproducibility: Code modularity should be improved to easily accommodate different datasets
- Optimization: Mitigating unfairness adds hyper-parameters and requires dealing with trade-offs between utility and fairness
- Comparability: Despite using similar datasets, evaluation settings are often different across papers
- Impact: Depending on the model, dataset, and task, mitigation procedures do not always substantially reduce unfairness
BibTeX
@inproceedings{boratto2022consumer,
author = {Boratto, Ludovico and Fenu, Gianni and Marras, Mirko and Medda, Giacomo},
title = {Consumer Fairness in Recommender Systems: Contextualizing Definitions and Mitigations},
booktitle = {Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022},
pages = {552--566},
year = {2022},
doi = {10.1007/978-3-030-99736-6_37},
publisher = {Springer}
}