Date: 7 April
Time: 1 p.m.
Venue: PA2 Amphitheatre, Mathematics Building, Alameda Campus
Speaker: José Pombal (Sword Health)
Title: “Zero-shot Benchmarking: Flexible and Scalable Automatic Evaluation of LLMs”
Abstract:
As language models improve and become capable of performing more complex tasks across modalities, evaluating them automatically becomes increasingly challenging. Developing strong and robust task-specific automatic metrics gets harder, and human-annotated test sets — which are expensive to create — saturate more quickly. A compelling alternative is to design reliable strategies to automate the creation of test data and evaluation, but previous attempts either rely on pre-existing data, or focus solely on individual tasks. We present Zero-shot Benchmarking (ZSB), a framework for creating high-quality benchmarks for any task by leveraging language models for both synthetic test data creation and evaluation. ZSB is simple and flexible: it requires only the creation of a prompt for data generation and one for evaluation; it is scalable to tasks and languages where collecting real-world data is costly or impractical; it is model-agnostic, allowing the creation of increasingly challenging benchmarks as models improve. To assess the effectiveness of our framework, we create benchmarks for five text-only tasks and a multi-modal one: general capabilities in four languages (English, Chinese, French, and Korean), translation, and general vision-language capabilities in English. We then rank a broad range of open and closed systems on our benchmarks. ZSB rankings consistently correlate strongly with human rankings, outperforming widely-adopted standard benchmarks. Through ablations, we find that strong benchmarks can be created with open models, and that judge model size and dataset variety are crucial drivers of performance. We release all our benchmarks, and code to reproduce our experiments and to produce new benchmarks.
Speaker bio:
José Pombal is a Senior Research Scientist at Sword Health and a PhD student at Instituto Superior Técnico, Universidade de Lisboa, with his research focusing on the automatic evaluation of LLMs and their application to mental health therapy.
Priberam is a member of the IST Spin-Off Community®.
The Priberam Machine Learning Lunch Seminars are free of charge. Prior registration is required.
