Date: March 12
Hora: 1 p.m.
Venue: Room EA4, North Tower, Alameda Campus
Speaker: Ricardo Rei (Unbabel)
Title: “From COMET to XCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection”
Abstract:
The transition from COMET to XCOMET marks a significant evolution in machine translation evaluation methodologies. COMET, a neural framework renowned for its prowess in training multilingual machine translation evaluation models, has long been celebrated for its state-of-the-art correlation with human judgments. Leveraging breakthroughs in cross-lingual pretrained language modeling, COMET excels in generating adaptable evaluation models capable of accurately predicting translation quality by assimilating information from both source input and target-language references. However, the landscape of machine translation evaluation is rapidly evolving, propelled by the emergence of generative large language models (LLMs) and the demand for more granular error analysis. In response to these challenges, XCOMET emerges as a groundbreaking solution, aiming to bridge the gap between traditional learned metrics and the evolving demands of the field. Unlike its predecessor, XCOMET integrates both sentence-level evaluation and error span detection capabilities, offering deeper insights into translation errors and their severity. In this presentation, we delve into the conceptualization and implementation of XCOMET, highlighting its novel features and advancements over COMET. By combining state-of-the-art sentence-level evaluation with error span detection, XCOMET enriches the quality assessment process, providing valuable insights into the nature and severity of translation errors. Through rigorous analysis and stress tests, we showcase the robustness of XCOMET in identifying critical errors and hallucinations, solidifying its position as a pioneering solution in the realm of machine translation evaluation.
Speaker bio:
Ricardo Rei is a senior research scientist at Unbabel, specializing in machine translation and natural language processing. He is set to complete his Ph.D. in April, which has been a collaborative effort between Unbabel, INESC-ID/Tecnico, and CMU University. His doctoral research has been centered on machine translation evaluation, and he is the main developer behind the COMET evaluation framework, which has become the industry standard metric for assessing machine translation quality. With a keen interest in advancing the capabilities of multilingual large language models (LLMs), he has been at the forefront of research and development in this domain. When not immersed in research, Ricardo enjoys maintaining an active lifestyle, often found at the gym or riding the waves while surfing—a passion he has pursued since he was nine.
Priberam is a member of the IST Spin-Off Community®.
The Priberam Machine Learning Lunch Seminars are free of charge. Prior registration is required.
