Data Contamination Report from the 2024 CONDA Shared Task

Oscar Sainz; Iker García-Ferrero; Alon Jacovi; Jon Ander Campos; Yanai Elazar; Eneko Agirre; Yoav Goldberg; Wei Lin Chen; Jenny Chim; Leshem Choshen; Luca D’Amico-Wong; Melissa Dell; Run Ze Fan; Shahriar Golchin; Yucheng Li; Pengfei Liu; Bhavish Pahwa; Ameya Prabhu; Suryansh Sharma; Emily Silcock; Kateryna Solonko; David Stap; Mihai Surdeanu; Yu Min Tseng; Vishaal Udandarao; Zengzhi Wang; Ruijie Xu; Jinglin Yang

Data Contamination Report from the 2024 CONDA Shared Task

Oscar Sainz, Iker García-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei Lin Chen, Jenny Chim, Leshem Choshen, Luca D’Amico-Wong, Melissa Dell, Run Ze Fan, Shahriar Golchin, Yucheng Li, Pengfei Liu, Bhavish Pahwa, Ameya Prabhu, Suryansh Sharma, Emily SilcockKateryna Solonko, David Stap, Mihai Surdeanu, Yu Min Tseng, Vishaal Udandarao, Zengzhi Wang, Ruijie Xu, Jinglin Yang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform.¹ The platform continues to be online, open to contributions from the community.

Original language	English (US)
Title of host publication	CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop
Editors	Oscar Sainz, Iker Garcia Ferrero, Eneko Agirre, Jon Ander Campos, Alon Jacovi, Yanai Elazar, Yoav Goldberg
Publisher	Association for Computational Linguistics (ACL)
Pages	41-56
Number of pages	16
ISBN (Electronic)	9798891761353
State	Published - 2024
Event	1st Data Contamination Workshop, CONDA 2024 - Bangkok, Thailand Duration: Aug 16 2024 → …

Publication series

Name	CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop

Conference

Conference	1st Data Contamination Workshop, CONDA 2024
Country/Territory	Thailand
City	Bangkok
Period	8/16/24 → …

ASJC Scopus subject areas

Language and Linguistics
Computer Science Applications
Linguistics and Language

Cite this

Sainz, O., García-Ferrero, I., Jacovi, A., Campos, J. A., Elazar, Y., Agirre, E., Goldberg, Y., Chen, W. L., Chim, J., Choshen, L., D’Amico-Wong, L., Dell, M., Fan, R. Z., Golchin, S., Li, Y., Liu, P., Pahwa, B., Prabhu, A., Sharma, S., ... Yang, J. (2024). Data Contamination Report from the 2024 CONDA Shared Task. In O. Sainz, I. G. Ferrero, E. Agirre, J. A. Campos, A. Jacovi, Y. Elazar, & Y. Goldberg (Eds.), CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop (pp. 41-56). (CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop). Association for Computational Linguistics (ACL).

Data Contamination Report from the 2024 CONDA Shared Task. / Sainz, Oscar; García-Ferrero, Iker; Jacovi, Alon et al.
CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop. ed. / Oscar Sainz; Iker Garcia Ferrero; Eneko Agirre; Jon Ander Campos; Alon Jacovi; Yanai Elazar; Yoav Goldberg. Association for Computational Linguistics (ACL), 2024. p. 41-56 (CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Sainz, O, García-Ferrero, I, Jacovi, A, Campos, JA, Elazar, Y, Agirre, E, Goldberg, Y, Chen, WL, Chim, J, Choshen, L, D’Amico-Wong, L, Dell, M, Fan, RZ, Golchin, S, Li, Y, Liu, P, Pahwa, B, Prabhu, A, Sharma, S, Silcock, E, Solonko, K, Stap, D, Surdeanu, M, Tseng, YM, Udandarao, V, Wang, Z, Xu, R & Yang, J 2024, Data Contamination Report from the 2024 CONDA Shared Task. in O Sainz, IG Ferrero, E Agirre, JA Campos, A Jacovi, Y Elazar & Y Goldberg (eds), CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop. CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop, Association for Computational Linguistics (ACL), pp. 41-56, 1st Data Contamination Workshop, CONDA 2024, Bangkok, Thailand, 8/16/24.

Sainz O, García-Ferrero I, Jacovi A, Campos JA, Elazar Y, Agirre E et al. Data Contamination Report from the 2024 CONDA Shared Task. In Sainz O, Ferrero IG, Agirre E, Campos JA, Jacovi A, Elazar Y, Goldberg Y, editors, CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2024. p. 41-56. (CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop).

Sainz, Oscar ; García-Ferrero, Iker ; Jacovi, Alon et al. / Data Contamination Report from the 2024 CONDA Shared Task. CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop. editor / Oscar Sainz ; Iker Garcia Ferrero ; Eneko Agirre ; Jon Ander Campos ; Alon Jacovi ; Yanai Elazar ; Yoav Goldberg. Association for Computational Linguistics (ACL), 2024. pp. 41-56 (CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop).

@inproceedings{01c7eafdadee40dbb0c6d50a1af7719a,

title = "Data Contamination Report from the 2024 CONDA Shared Task",

abstract = "The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform.1 The platform continues to be online, open to contributions from the community.",

author = "Oscar Sainz and Iker Garc{\'i}a-Ferrero and Alon Jacovi and Campos, {Jon Ander} and Yanai Elazar and Eneko Agirre and Yoav Goldberg and Chen, {Wei Lin} and Jenny Chim and Leshem Choshen and Luca D{\textquoteright}Amico-Wong and Melissa Dell and Fan, {Run Ze} and Shahriar Golchin and Yucheng Li and Pengfei Liu and Bhavish Pahwa and Ameya Prabhu and Suryansh Sharma and Emily Silcock and Kateryna Solonko and David Stap and Mihai Surdeanu and Tseng, {Yu Min} and Vishaal Udandarao and Zengzhi Wang and Ruijie Xu and Jinglin Yang",

note = "Publisher Copyright: {\textcopyright}2024 Association for Computational Linguistics.; 1st Data Contamination Workshop, CONDA 2024 ; Conference date: 16-08-2024",

year = "2024",

language = "English (US)",

series = "CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "41--56",

editor = "Oscar Sainz and Ferrero, {Iker Garcia} and Eneko Agirre and Campos, {Jon Ander} and Alon Jacovi and Yanai Elazar and Yoav Goldberg",

booktitle = "CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop",

}

TY - GEN

T1 - Data Contamination Report from the 2024 CONDA Shared Task

AU - Sainz, Oscar

AU - García-Ferrero, Iker

AU - Jacovi, Alon

AU - Campos, Jon Ander

AU - Elazar, Yanai

AU - Agirre, Eneko

AU - Goldberg, Yoav

AU - Chen, Wei Lin

AU - Chim, Jenny

AU - Choshen, Leshem

AU - D’Amico-Wong, Luca

AU - Dell, Melissa

AU - Fan, Run Ze

AU - Golchin, Shahriar

AU - Li, Yucheng

AU - Liu, Pengfei

AU - Pahwa, Bhavish

AU - Prabhu, Ameya

AU - Sharma, Suryansh

AU - Silcock, Emily

AU - Solonko, Kateryna

AU - Stap, David

AU - Surdeanu, Mihai

AU - Tseng, Yu Min

AU - Udandarao, Vishaal

AU - Wang, Zengzhi

AU - Xu, Ruijie

AU - Yang, Jinglin

PY - 2024

Y1 - 2024

N2 - The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform.1 The platform continues to be online, open to contributions from the community.

AB - The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform.1 The platform continues to be online, open to contributions from the community.

UR - http://www.scopus.com/inward/record.url?scp=85204282889&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85204282889&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85204282889

T3 - CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop

SP - 41

EP - 56

BT - CONDA 2024 - 1st Data Contamination Workshop, Proceedings of the Workshop

A2 - Sainz, Oscar

A2 - Ferrero, Iker Garcia

A2 - Agirre, Eneko

A2 - Campos, Jon Ander

A2 - Jacovi, Alon

A2 - Elazar, Yanai

A2 - Goldberg, Yoav

PB - Association for Computational Linguistics (ACL)

T2 - 1st Data Contamination Workshop, CONDA 2024

Y2 - 16 August 2024

ER -

Data Contamination Report from the 2024 CONDA Shared Task

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this