Refining Chinese–English bidirectional interpreting assessment: Automated assessment based on fidelity and target language quality

This study proposed an automated interpreting quality assessment (IQA) approach for bidirectional Chinese–English interpreting as an alternative to the conventional human-mediated IQA, which is often scrutinized due to concerns over its reliability and consistency. Different from previous research on automated IQA, which typically focuses on a single quality criterion (e.g., Lu & Han, 2025; Wu, 2021) or relies on resource-intensive reference translations (e.g., Wang et al., 2022), this study evaluated both fidelity and target language (TL) quality. Specifically, this study prioritized fidelity as the dominant criterion, with TL quality serving as a supporting one, in alignment with prevalent human-mediated IQA criteria. The study analyzed 400 bidirectional student interpretations sourced from the Parallel Corpus of Chinese EFL Learners (PACCEL; Wen & Wang, 2008). Each interpretation was accompanied by a human rating result, and the goal of this study was to predict these human ratings. Fidelity was evaluated using the Language-Agnostic BERT Sentence Embedding (LaBSE; Feng et al., 2021) model to obtain the cross-language semantic similarity score. TL quality was assessed using the Tool for the Automatic Analysis of Lexical Sophistication (TAALES 2.2; Kyle et al., 2018) and Coh-Metrix 3.0 (Graesser et al., 2004). Two predicative models were developed: Model 1, based solely on fidelity indicated by cross-language semantic similarity scores, and Model 2, incorporating both fidelity and TL quality. Prediction performance was evaluated using t-tests, random forests, correlation analyses, and regression models. Results indicated that both models showed fair prediction results, with Model 2 showing improved prediction performance over Model 1 (i.e., Chinese-to-English: 0.776 vs. 0.711; English-to-Chinese: 0.781 vs. 0.762). The improvement was particularly notable for intermediate-level interpretations, with the correlation coefficient (r) increasing by 0.120 in Chinese-to-English interpreting (from 0.519 to 0.639) and by 0.130 in English-to-Chinese interpreting (from 0.531 to 0.661). Furthermore, differences in prediction performance were observed between interpreting directions, with generally better results in English-to-Chinese interpreting. These encouraging, albeit preliminary, findings suggest the potential for developing a comprehensive and more reliable automated assessment for interpreting.