TY - GEN
T1 - TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration
AU - Thayaparan, Mokanarangan
AU - Valentino, Marco
AU - Jansen, Peter
AU - Ustalov, Dmitry
N1 - Publisher Copyright:
© 2021 Association for Computational Linguistics.
PY - 2021
Y1 - 2021
N2 - The Shared Task on Multi-Hop Inference for Explanation Regeneration asks participants to compose large multi-hop explanations to questions by assembling large chains of facts from a supporting knowledge base. While previous editions of this shared task aimed to evaluate explanatory completeness - finding a set of facts that form a complete inference chain, without gaps, to arrive from question to correct answer, this 2021 instantiation concentrates on the subtask of determining relevance in large multi-hop explanations. To this end, this edition of the shared task makes use of a large set of approximately 250k manual explanatory relevancy ratings that augment the 2020 shared task data. In this summary paper, we describe the details of the explanation regeneration task, the evaluation data, and the participating systems. Additionally, we perform a detailed analysis of participating systems, evaluating various aspects involved in the multi-hop inference process. The best performing system achieved an NDCG of 0.82 on this challenging task, substantially increasing performance over baseline methods by 32%, while also leaving significant room for future improvement.
AB - The Shared Task on Multi-Hop Inference for Explanation Regeneration asks participants to compose large multi-hop explanations to questions by assembling large chains of facts from a supporting knowledge base. While previous editions of this shared task aimed to evaluate explanatory completeness - finding a set of facts that form a complete inference chain, without gaps, to arrive from question to correct answer, this 2021 instantiation concentrates on the subtask of determining relevance in large multi-hop explanations. To this end, this edition of the shared task makes use of a large set of approximately 250k manual explanatory relevancy ratings that augment the 2020 shared task data. In this summary paper, we describe the details of the explanation regeneration task, the evaluation data, and the participating systems. Additionally, we perform a detailed analysis of participating systems, evaluating various aspects involved in the multi-hop inference process. The best performing system achieved an NDCG of 0.82 on this challenging task, substantially increasing performance over baseline methods by 32%, while also leaving significant room for future improvement.
UR - http://www.scopus.com/inward/record.url?scp=85134069877&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85134069877&partnerID=8YFLogxK
U2 - 10.18653/v1/11.textgraphs-1.17
DO - 10.18653/v1/11.textgraphs-1.17
M3 - Conference contribution
AN - SCOPUS:85134069877
T3 - TextGraphs 2021 - Graph-Based Methods for Natural Language Processing, Proceedings of the 15th Workshop - in conjunction with the 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2021
SP - 156
EP - 165
BT - TextGraphs 2021 - Graph-Based Methods for Natural Language Processing, Proceedings of the 15th Workshop - in conjunction with the 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2021
A2 - Panchenko, Alexander
A2 - Malliaros, Fragkiskos D.
A2 - Logacheva, Varvara
A2 - Jana, Abhik
A2 - Ustalov, Dmitry
A2 - Jansen, Peter
PB - Association for Computational Linguistics (ACL)
T2 - 15th Workshop on Graph-Based Methods for Natural Language Processing, TextGraphs 2021
Y2 - 11 June 2021
ER -