[TMP-102] Promoting Fairness and Diversity in Speech Datasets for Affective Computing
This project addresses biases in AI speech datasets by reviewing literature, analyzing dataset limitations, and promoting diversity to improve inclusivity and effectiveness.
As AI-powered devices and software become integral to daily life, it is crucial to prevent the perpetuation of stereotypes and biases related to gender, age, race, and other characteristics vulnerable to discrimination. Current practices for creating datasets used in machine learning models have well-documented limitations, including a lack of diversity. These shortcomings, combined with the homogeneity of stakeholders in AI development, exacerbate structural biases. Addressing this issue requires the AI community to promote diversity, understand its impact, and develop inclusive solutions. Speech, a natural mode of human communication, conveys nuanced emotional and contextual information beyond text. However, many speech datasets lack diversity in speakers, leading to biases in speech recognition systems, especially for varied accents, dialects, and speech patterns. Additional issues include narrow contexts, limited recording scales, and data quality concerns, which hinder the effectiveness of AI in applications like virtual assistants, mental health diagnostics, and emotion recognition. This micro-project aims to enhance future speech datasets and improve the use of existing ones. We will review literature and existing datasets to identify essential features for unbiased and inclusive data. Additionally, a meta-analysis will critically evaluate dataset limitations and their presentation in scientific studies, enabling a comprehensive assessment of the field.
In this micro-project, we addressed the domain of speech datasets for mental health and neurological disorders. We created a set of 7 desiderata for building these datasets, distilled it into a checklist of 20 elements that can be used as a tool for analysis of existing works and as guidance for future works, and finally surveyed existing literature to analyze and discuss current practices. Our set of desiderata is the first to specifically address this domain and considers both aspects that are relevant in terms of ethics and societal impact, such as “Fairness, Bias, and Diversity”, but also aspects that are more technical and domain-specific, such as the details of the recording and the involvement of medical experts in the study.
In our survey of existing literature, we identified key areas for improvement in resource creation and use. For example, several of the examined papers do not report on informed consent and accountability. Our findings highlighted the importance of involving experts from several different disciplines (e.g., computer science, medicine, social science, and law) when conducting studies in such a critical domain. These results also confirm the importance of the dissemination of principles and best practices across different disciplines.
Tangible Outcomes
- [arxiv] [under review] 1.A pre-print currently under review: Mancini, E., Tanevska, A., Galassi, A., Galatolo, A., Ruggeri, F., & Torroni, P. (2024). Promoting Fairness and Diversity in Speech Datasets for Mental Health and Neurological Disorders Research. arXiv preprint arXiv:2406.04116 (under review in JAIR (journal of artificial intelligence research))
https://arxiv.org/abs/2406.04116 - A GitHub repository with detailed analysis of literature Detailed analysis of containing 36 existing datasets and papers according to our desiderata and checklist
https://github.com/nlp-unibo/ethical-survey-speech - Invited talk “Towards an Ethical and Human-centric Artificial Intelligence: two case studies on fairness in Dialogue Systems and Speech Datasets”, at “2nd Workshop on Inside the Ethics of AI Awareness”, November 11th 2024, Uppsala, organized as part of the Horizon Europe project SymAware
Partners
- University of Bologna, Andrea Galassi, a.galasi@unibo.it
- Uppsala University, Ana Tanevska, ana.tanevska@it.uu.s