Although large language models can translate some historical languages surprisingly well, their usefulness in digital humanities workflows is limited by the lack of reliable evaluation. We investigate whether existing automatic evaluation metrics developed for modern languages are reliable in this setting, using translation from Classical Chinese to English as a test case. We introduce a diagnostic framework based on minimal pairs capturing error types salient in scholarly use, probing both reference-based and reference-free metrics for error sensitivity and tolerance to valid variation. We find that all metrics exhibit blind spots, however MetricX24 performs best overall. Our findings highlight the need for more robust and interpretable metrics for historically and culturally distinct translation settings.
@inproceedings{quinjica2026classical,title={Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?},author={Quinjica, Osvaldo and Bennett, Eric and Yang, Xinchen and Schonebaum, Andrew and Carpuat, Marine},booktitle={Conference on Language Modeling (COLM)},year={2026},}
Best Paper Award at the AfricaNLP 2024 workshop (ICLR 2024).
In recent years, the development of pre-trained language models (PLMs) has gained momentum, showcasing their capacity to transcend linguistic barriers and facilitate knowledge transfer across diverse languages. However, this progress has predominantly bypassed the inclusion of very-low resource languages, creating a notable void in the multilingual landscape. This paper addresses this gap by introducing four tailored PLMs specifically finetuned for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT) approach. In this paper, we survey the role of informed embedding initialization and synthetic data in enhancing the performance of MAFT models in downstream tasks. We improve baseline over SOTA AfroXLMR-base (developed through MAFT) and OFA (an effective embedding initialization) by 12.3 and 3.8 points respectively.
@inproceedings{quinjica2024angofa,title={AngOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model},author={Quinjica, Osvaldo and Adelani, David Ifeoluwa},booktitle={ICLR 2024 Workshop on African Natural Language Processing (AfricaNLP)},year={2024},}
@inproceedings{pan2022cauchy,title={Generalization of Cauchy-Residue Formula},author={Pan, Yuzhou and Quinjica, Osvaldo L. and Zhu, Ranwen and Qian, Yichen},booktitle={2nd International Conference on Applied Mathematics, Modelling, and Intelligent Computing (CAMMIC 2022)},series={Proceedings of SPIE},volume={12259},pages={418--428},publisher={SPIE},year={2022},doi={10.1117/12.2639346},}