Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge Retrieval from Foundation Language Models

We assess the ability of foundation models to recall encyclopedic knowledge across a wide range of linguistic contexts. To support this, we: 1) produce a 20-language dataset that contains 303k factual associations paired with counterfactuals, 2) evaluate 5 models in a multilingual test, and 3) benchmark a diverse set of 24 models in an English-only test. Meta's LLaMA achieves the highest scores in both multilingual and English-only evaluations. Yet, an analysis of LLaMA's errors reveals significant limitations in its ability to recall facts in languages other than English, plus difficulties related to the location and gender of fact subjects. Overall, our findings suggest that today's foundation models are far from polyglots. Our research artifacts (code, dataset, and analysis) are entirely open-source, empowering future researchers to perform their own experiments on future foundation language models.

This work was published following our capstone; you can find our paper in EMNLP:

Tim Schott, Daniel Furman, and Shreshta Bhat. 2023. Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11238–11253, Singapore. Association for Computational Linguistics.

Fact/Fake illustration by dictionary.com