Elliot: A Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation
Author(s)
Date Issued
2021
Type
conferenceObject
Start Page
2405
End Page
2414
Abstract
Recommender Systems have shown to be an effective way to alleviate the over-choice problem and provide accurate and tailored recommendations. However, the impressive number of proposed recommendation algorithms, splitting strategies, evaluation protocols, metrics, and tasks, has made rigorous experimental evaluation particularly challenging. Puzzled and frustrated by the continuous recreation of appropriate evaluation benchmarks, experimental pipelines, hyperparameter optimization, and evaluation procedures, we have developed an exhaustive framework to address such needs. Elliot is a comprehensive recommendation framework that aims to run and reproduce an entire experimental pipeline by processing a simple configuration file. The framework loads, filters, and splits the data considering a vast set of strategies (13 splitting methods and 8 filtering approaches, from temporal training-test splitting to nested K-folds Cross-Validation). Elliot(https://github.com/sisinflab/elliot) optimizes hyperparameters (51 strategies) for several recommendation algorithms (50), selects the best models, compares them with the baselines providing intra-model statistics, computes metrics (36) spanning from accuracy to beyond-accuracy, bias, and fairness, and conducts statistical analysis (Wilcoxon and Paired t-test).
File(s)![Thumbnail Image]()
Name
SIGIR2021_Elliot_RecSys_Framework.pdf
Description
Green Open Access
Size
755.98 KB
Format
Adobe PDF
Checksum (MD5)
26a863f1a960c86965d3c2159d394cda
Conference(s)
SIGIR 2021 - 44th International ACM SIGIR Conference on Research and Development in Information Retrieval