Language models for data extraction and risk of bias assessment in complementary medicine

Honghao Lai; Jiayi Liu; Chunyang Bai; Hui Liu; Bei Pan; Xufei Luo; Liangying Hou; Weilong Zhao; Danni Xia; Jinhui Tian; Yaolong Chen; Lu Zhang; Janne Estill; Jie Liu; Xing Liao; Nannan Shi; Xin Sun; Hongcai Shang; Zhaoxiang Bian; Kehu Yang; Luqi Huang; Long Ge; ADVANCED Working Group

doi:10.1038/s41746-025-01457-w

Language models for data extraction and risk of bias assessment in complementary medicine

Honghao Lai
, Jiayi Liu
, Chunyang Bai
, Hui Liu
, Bei Pan
, Xufei Luo
, Liangying Hou
, Weilong Zhao
, Danni Xia
, Jinhui Tian
, Yaolong Chen
, Lu Zhang
, Janne Estill
, Jie Liu
, Xing Liao
, Nannan Shi
, Xin Sun
, Hongcai Shang
, Zhaoxiang Bian
, Kehu Yang

Luqi Huang^*, Long Ge^*, ADVANCED Working Group

^*Corresponding author for this work

Research output: Contribution to journal › Journal article › peer-review

27 Citations (Scopus)

Abstract

Large language models (LLMs) have the potential to enhance evidence synthesis efficiency and accuracy. This study assessed LLM-only and LLM-assisted methods in data extraction and risk of bias assessment for 107 trials on complementary medicine. Moonshot-v1-128k and Claude-3.5-sonnet achieved high accuracy (≥95%), with LLM-assisted methods performing better (≥97%). LLM-assisted methods significantly reduced processing time (14.7 and 5.9 min vs. 86.9 and 10.4 min for conventional methods). These findings highlight LLMs’ potential when integrated with human expertise.

Original language	English
Article number	74
Number of pages	8
Journal	npj Digital Medicine
Volume	8
Issue number	1
DOIs	https://doi.org/10.1038/s41746-025-01457-w
Publication status	Published - 31 Jan 2025

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Access to Document

10.1038/s41746-025-01457-w

Cite this

Lai, H., Liu, J., Bai, C., Liu, H., Pan, B., Luo, X., Hou, L., Zhao, W., Xia, D., Tian, J., Chen, Y., Zhang, L., Estill, J., Liu, J., Liao, X., Shi, N., Sun, X., Shang, H., Bian, Z., ... ADVANCED Working Group (2025). Language models for data extraction and risk of bias assessment in complementary medicine. npj Digital Medicine, 8(1), Article 74. https://doi.org/10.1038/s41746-025-01457-w

@article{d14edc858fe941b1b8259486202120f2,

title = "Language models for data extraction and risk of bias assessment in complementary medicine",

abstract = "Large language models (LLMs) have the potential to enhance evidence synthesis efficiency and accuracy. This study assessed LLM-only and LLM-assisted methods in data extraction and risk of bias assessment for 107 trials on complementary medicine. Moonshot-v1-128k and Claude-3.5-sonnet achieved high accuracy (≥95\%), with LLM-assisted methods performing better (≥97\%). LLM-assisted methods significantly reduced processing time (14.7 and 5.9 min vs. 86.9 and 10.4 min for conventional methods). These findings highlight LLMs{\textquoteright} potential when integrated with human expertise.",

author = "Honghao Lai and Jiayi Liu and Chunyang Bai and Hui Liu and Bei Pan and Xufei Luo and Liangying Hou and Weilong Zhao and Danni Xia and Jinhui Tian and Yaolong Chen and Lu Zhang and Janne Estill and Jie Liu and Xing Liao and Nannan Shi and Xin Sun and Hongcai Shang and Zhaoxiang Bian and Kehu Yang and Luqi Huang and Long Ge and \{ADVANCED Working Group\}",

note = "This study was jointly supported by the Fundamental Research Funds for the Central Universities (No. lzujbky-2024-oy11), the National Natural Science Foundation of China (No. 82204931) and the Scientific and Technological Innovation Project of the China Academy of Chinese Medical Sciences (No. CI2021A05502). Funding info provided by Prof. William Cheung: Supported by the research project {"}Artificial Intelligence and Big Data Analytics for Social Good{"} (RMGS2021\_8\_06). {\textcopyright} The Author(s) 2025",

year = "2025",

month = jan,

day = "31",

doi = "10.1038/s41746-025-01457-w",

language = "English",

volume = "8",

journal = "npj Digital Medicine",

issn = "2398-6352",

publisher = "Nature Publishing Group",

number = "1",

}

Lai, H, Liu, J, Bai, C, Liu, H, Pan, B, Luo, X, Hou, L, Zhao, W, Xia, D, Tian, J, Chen, Y, Zhang, L, Estill, J, Liu, J, Liao, X, Shi, N, Sun, X, Shang, H, Bian, Z, Yang, K, Huang, L, Ge, L & ADVANCED Working Group 2025, 'Language models for data extraction and risk of bias assessment in complementary medicine', npj Digital Medicine, vol. 8, no. 1, 74. https://doi.org/10.1038/s41746-025-01457-w

TY - JOUR

T1 - Language models for data extraction and risk of bias assessment in complementary medicine

AU - Lai, Honghao

AU - Liu, Jiayi

AU - Bai, Chunyang

AU - Liu, Hui

AU - Pan, Bei

AU - Luo, Xufei

AU - Hou, Liangying

AU - Zhao, Weilong

AU - Xia, Danni

AU - Tian, Jinhui

AU - Chen, Yaolong

AU - Zhang, Lu

AU - Estill, Janne

AU - Liu, Jie

AU - Liao, Xing

AU - Shi, Nannan

AU - Sun, Xin

AU - Shang, Hongcai

AU - Bian, Zhaoxiang

AU - Yang, Kehu

AU - Huang, Luqi

AU - Ge, Long

AU - ADVANCED Working Group

N1 - This study was jointly supported by the Fundamental Research Funds for the Central Universities (No. lzujbky-2024-oy11), the National Natural Science Foundation of China (No. 82204931) and the Scientific and Technological Innovation Project of the China Academy of Chinese Medical Sciences (No. CI2021A05502). Funding info provided by Prof. William Cheung: Supported by the research project "Artificial Intelligence and Big Data Analytics for Social Good" (RMGS2021_8_06). © The Author(s) 2025

PY - 2025/1/31

Y1 - 2025/1/31

N2 - Large language models (LLMs) have the potential to enhance evidence synthesis efficiency and accuracy. This study assessed LLM-only and LLM-assisted methods in data extraction and risk of bias assessment for 107 trials on complementary medicine. Moonshot-v1-128k and Claude-3.5-sonnet achieved high accuracy (≥95%), with LLM-assisted methods performing better (≥97%). LLM-assisted methods significantly reduced processing time (14.7 and 5.9 min vs. 86.9 and 10.4 min for conventional methods). These findings highlight LLMs’ potential when integrated with human expertise.

AB - Large language models (LLMs) have the potential to enhance evidence synthesis efficiency and accuracy. This study assessed LLM-only and LLM-assisted methods in data extraction and risk of bias assessment for 107 trials on complementary medicine. Moonshot-v1-128k and Claude-3.5-sonnet achieved high accuracy (≥95%), with LLM-assisted methods performing better (≥97%). LLM-assisted methods significantly reduced processing time (14.7 and 5.9 min vs. 86.9 and 10.4 min for conventional methods). These findings highlight LLMs’ potential when integrated with human expertise.

UR - https://www.scopus.com/pages/publications/105002937184

U2 - 10.1038/s41746-025-01457-w

DO - 10.1038/s41746-025-01457-w

M3 - Journal article

C2 - 39890970

SN - 2398-6352

VL - 8

JO - npj Digital Medicine

JF - npj Digital Medicine

IS - 1

M1 - 74

ER -

Language models for data extraction and risk of bias assessment in complementary medicine

Abstract

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this