MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video

Xiaoqing Guo; Qianhui Men; J. Alison Noble

doi:10.1007/978-3-031-72083-3_63

MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video

Xiaoqing Guo^*, Qianhui Men, J. Alison Noble

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

Abstract

We present the first automated multimodal summary generation system, MMSummary, for medical imaging video, particularly with a focus on fetal ultrasound analysis. Imitating the examination process performed by a human sonographer, MMSummary is designed as a three-stage pipeline, progressing from keyframe detection to keyframe captioning and finally anatomy segmentation and measurement. In the keyframe detection stage, an innovative automated workflow is proposed to progressively select a concise set of keyframes, preserving sufficient video information without redundancy. Subsequently, we adapt a large language model to generate meaningful captions for fetal ultrasound keyframes in the keyframe captioning stage. If a keyframe is captioned as fetal biometry, the segmentation and measurement stage estimates biometric parameters by segmenting the region of interest according to the textual prior. The MMSummary system provides comprehensive summaries for fetal ultrasound examinations and based on reported experiments is estimated to reduce scanning time by approximately 31.5%, thereby suggesting the potential to enhance clinical workflow efficiency.

Original language	English
Title of host publication	Medical Image Computing and Computer Assisted Intervention – MICCAI 2024
Subtitle of host publication	27th International Conference, Marrakesh, Morocco, October 6–10, 2024, Proceedings, Part IV
Editors	Marius George Linguraru, Qi Dou, Aasa Feragen, Stamatia Giannarou, Ben Glocker, Karim Lekadir, Julia A. Schnabel
Publisher	Springer Nature
Pages	678-688
Number of pages	11
ISBN (Electronic)	9783031720833
ISBN (Print)	9783031720826
DOIs	https://doi.org/10.1007/978-3-031-72083-3_63
Publication status	Published - 13 Oct 2024
Event	27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024 - Marrakesh, Morocco Duration: 6 Oct 2024 → 10 Oct 2024 https://link.springer.com/book/10.1007/978-3-031-72083-3 (Conference Proceedings (Part IV)) https://link.springer.com/book/10.1007/978-3-031-72111-3 (Conference Proceedings (Part VIII))

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	15004
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Name	MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention
Publisher	Springer

Conference

Conference	27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024
Country/Territory	Morocco
City	Marrakesh
Period	6/10/24 → 10/10/24
Internet address	https://link.springer.com/book/10.1007/978-3-031-72083-3 (Conference Proceedings (Part IV)) https://link.springer.com/book/10.1007/978-3-031-72111-3 (Conference Proceedings (Part VIII))

Scopus Subject Areas

Theoretical Computer Science
General Computer Science

User-Defined Keywords

Multimodal summary
Video analysis
Fetal ultrasound

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-031-72083-3_63

Cite this

Guo, X., Men, Q., & Noble, J. A. (2024). MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video. In M. G. Linguraru, Q. Dou, A. Feragen, S. Giannarou, B. Glocker, K. Lekadir, & J. A. Schnabel (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2024: 27th International Conference, Marrakesh, Morocco, October 6–10, 2024, Proceedings, Part IV (pp. 678-688). (Lecture Notes in Computer Science; Vol. 15004), (MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention). Springer Nature. https://doi.org/10.1007/978-3-031-72083-3_63

Guo, Xiaoqing ; Men, Qianhui ; Noble, J. Alison. / MMSummary : Multimodal Summary Generation for Fetal Ultrasound Video. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024: 27th International Conference, Marrakesh, Morocco, October 6–10, 2024, Proceedings, Part IV. editor / Marius George Linguraru ; Qi Dou ; Aasa Feragen ; Stamatia Giannarou ; Ben Glocker ; Karim Lekadir ; Julia A. Schnabel. Springer Nature, 2024. pp. 678-688 (Lecture Notes in Computer Science). (MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention).

@inproceedings{e382a4dd3146449cb3781d0773d41f85,

title = "MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video",

abstract = "We present the first automated multimodal summary generation system, MMSummary, for medical imaging video, particularly with a focus on fetal ultrasound analysis. Imitating the examination process performed by a human sonographer, MMSummary is designed as a three-stage pipeline, progressing from keyframe detection to keyframe captioning and finally anatomy segmentation and measurement. In the keyframe detection stage, an innovative automated workflow is proposed to progressively select a concise set of keyframes, preserving sufficient video information without redundancy. Subsequently, we adapt a large language model to generate meaningful captions for fetal ultrasound keyframes in the keyframe captioning stage. If a keyframe is captioned as fetal biometry, the segmentation and measurement stage estimates biometric parameters by segmenting the region of interest according to the textual prior. The MMSummary system provides comprehensive summaries for fetal ultrasound examinations and based on reported experiments is estimated to reduce scanning time by approximately 31.5%, thereby suggesting the potential to enhance clinical workflow efficiency.",

keywords = "Multimodal summary, Video analysis, Fetal ultrasound",

author = "Xiaoqing Guo and Qianhui Men and Noble, {J. Alison}",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024 ; Conference date: 06-10-2024 Through 10-10-2024",

year = "2024",

month = oct,

day = "13",

doi = "10.1007/978-3-031-72083-3_63",

language = "English",

isbn = "9783031720826",

series = "Lecture Notes in Computer Science",

publisher = "Springer Nature",

pages = "678--688",

editor = "Linguraru, {Marius George} and Qi Dou and Aasa Feragen and Stamatia Giannarou and Ben Glocker and Karim Lekadir and Schnabel, {Julia A.}",

booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2024",

address = "Switzerland",

url = "https://link.springer.com/book/10.1007/978-3-031-72083-3, https://link.springer.com/book/10.1007/978-3-031-72111-3",

}

Guo, X, Men, Q & Noble, JA 2024, MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video. in MG Linguraru, Q Dou, A Feragen, S Giannarou, B Glocker, K Lekadir & JA Schnabel (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2024: 27th International Conference, Marrakesh, Morocco, October 6–10, 2024, Proceedings, Part IV. Lecture Notes in Computer Science, vol. 15004, MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer Nature, pp. 678-688, 27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024, Marrakesh, Morocco, 6/10/24. https://doi.org/10.1007/978-3-031-72083-3_63

MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video. / Guo, Xiaoqing; Men, Qianhui; Noble, J. Alison.
Medical Image Computing and Computer Assisted Intervention – MICCAI 2024: 27th International Conference, Marrakesh, Morocco, October 6–10, 2024, Proceedings, Part IV. ed. / Marius George Linguraru; Qi Dou; Aasa Feragen; Stamatia Giannarou; Ben Glocker; Karim Lekadir; Julia A. Schnabel. Springer Nature, 2024. p. 678-688 (Lecture Notes in Computer Science; Vol. 15004), (MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - MMSummary

T2 - 27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024

AU - Guo, Xiaoqing

AU - Men, Qianhui

AU - Noble, J. Alison

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

PY - 2024/10/13

Y1 - 2024/10/13

N2 - We present the first automated multimodal summary generation system, MMSummary, for medical imaging video, particularly with a focus on fetal ultrasound analysis. Imitating the examination process performed by a human sonographer, MMSummary is designed as a three-stage pipeline, progressing from keyframe detection to keyframe captioning and finally anatomy segmentation and measurement. In the keyframe detection stage, an innovative automated workflow is proposed to progressively select a concise set of keyframes, preserving sufficient video information without redundancy. Subsequently, we adapt a large language model to generate meaningful captions for fetal ultrasound keyframes in the keyframe captioning stage. If a keyframe is captioned as fetal biometry, the segmentation and measurement stage estimates biometric parameters by segmenting the region of interest according to the textual prior. The MMSummary system provides comprehensive summaries for fetal ultrasound examinations and based on reported experiments is estimated to reduce scanning time by approximately 31.5%, thereby suggesting the potential to enhance clinical workflow efficiency.

AB - We present the first automated multimodal summary generation system, MMSummary, for medical imaging video, particularly with a focus on fetal ultrasound analysis. Imitating the examination process performed by a human sonographer, MMSummary is designed as a three-stage pipeline, progressing from keyframe detection to keyframe captioning and finally anatomy segmentation and measurement. In the keyframe detection stage, an innovative automated workflow is proposed to progressively select a concise set of keyframes, preserving sufficient video information without redundancy. Subsequently, we adapt a large language model to generate meaningful captions for fetal ultrasound keyframes in the keyframe captioning stage. If a keyframe is captioned as fetal biometry, the segmentation and measurement stage estimates biometric parameters by segmenting the region of interest according to the textual prior. The MMSummary system provides comprehensive summaries for fetal ultrasound examinations and based on reported experiments is estimated to reduce scanning time by approximately 31.5%, thereby suggesting the potential to enhance clinical workflow efficiency.

KW - Multimodal summary

KW - Video analysis

KW - Fetal ultrasound

UR - http://www.scopus.com/inward/record.url?scp=85207643965&partnerID=8YFLogxK

UR - https://link.springer.com/book/10.1007/978-3-031-72083-3

U2 - 10.1007/978-3-031-72083-3_63

DO - 10.1007/978-3-031-72083-3_63

M3 - Conference proceeding

AN - SCOPUS:85207643965

SN - 9783031720826

T3 - Lecture Notes in Computer Science

SP - 678

EP - 688

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2024

A2 - Linguraru, Marius George

A2 - Dou, Qi

A2 - Feragen, Aasa

A2 - Giannarou, Stamatia

A2 - Glocker, Ben

A2 - Lekadir, Karim

A2 - Schnabel, Julia A.

PB - Springer Nature

Y2 - 6 October 2024 through 10 October 2024

ER -

Guo X, Men Q, Noble JA. MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video. In Linguraru MG, Dou Q, Feragen A, Giannarou S, Glocker B, Lekadir K, Schnabel JA, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2024: 27th International Conference, Marrakesh, Morocco, October 6–10, 2024, Proceedings, Part IV. Springer Nature. 2024. p. 678-688. (Lecture Notes in Computer Science). (MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention). doi: 10.1007/978-3-031-72083-3_63

MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video

Abstract

Publication series

Conference

Scopus Subject Areas

User-Defined Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this