Panoptic Video Scene Graph Generation

Jingkang Yang; Wenxuan Peng; Xiangtai Li; Zujin Guo; Liangyu Chen; Bo Li; Zheng Ma; Kaiyang Zhou; Wayne Zhang; Chen Change Loy; Ziwei Liu

doi:10.1109/CVPR52729.2023.01791

Panoptic Video Scene Graph Generation

Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, Ziwei Liu^*

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

Abstract

Towards building comprehensive real-world visual perception systems, we propose and study a new problem called panoptic scene graph generation (PVSG). PVSG is related to the existing video scene graph generation (VidSGG) problem, which focuses on temporal interactions between humans and objects localized with bounding boxes in videos. However, the limitation of bounding boxes in detecting non-rigid objects and backgrounds often causes VidSGG systems to miss key details that are crucial for comprehensive video understanding. In contrast, PVSG requires nodes in scene graphs to be grounded by more precise, pixel-level segmentation masks, which facilitate holistic scene understanding. To advance research in this new area, we contribute a high-quality PVSG dataset, which consists of 400 videos (289 third-person + 111 egocentric videos) with totally 150K frames labeled with panoptic segmentation masks as well as fine, temporal scene graphs. We also provide a variety of baseline methods and share useful design practices for future work.

Original language	English
Title of host publication	Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher	IEEE
Pages	18675-18685
Number of pages	11
ISBN (Electronic)	9798350301298
ISBN (Print)	9798350301304
DOIs	https://doi.org/10.1109/CVPR52729.2023.01791
Publication status	Published - 17 Jun 2023
Event	36th IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 - Vancouver, Canada Duration: 18 Jun 2023 → 22 Jun 2023 https://cvpr2023.thecvf.com/virtual/2023/index.html https://openaccess.thecvf.com/CVPR2023 https://cvpr2023.thecvf.com/virtual/2023/papers.html?filter=titles https://ieeexplore.ieee.org/xpl/conhome/10203037/proceeding

Publication series

Name	Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
ISSN (Print)	1063-6919
ISSN (Electronic)	2575-7075

Conference

Conference	36th IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
Country/Territory	Canada
City	Vancouver
Period	18/06/23 → 22/06/23
Internet address	https://cvpr2023.thecvf.com/virtual/2023/index.html https://openaccess.thecvf.com/CVPR2023 https://cvpr2023.thecvf.com/virtual/2023/papers.html?filter=titles https://ieeexplore.ieee.org/xpl/conhome/10203037/proceeding

Access to Document

10.1109/CVPR52729.2023.01791

https://openaccess.thecvf.com/content/CVPR2023/papers/Yang_Panoptic_Video_Scene_Graph_Generation_CVPR_2023_paper.pdf

Cite this

Yang, J., Peng, W., Li, X., Guo, Z., Chen, L., Li, B., Ma, Z., Zhou, K., Zhang, W., Loy, C. C., & Liu, Z. (2023). Panoptic Video Scene Graph Generation. In Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 18675-18685). ( Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)). IEEE. https://doi.org/10.1109/CVPR52729.2023.01791

@inproceedings{75ed2d7579c44ad2be79a39e6e5812d0,

title = "Panoptic Video Scene Graph Generation",

abstract = "Towards building comprehensive real-world visual perception systems, we propose and study a new problem called panoptic scene graph generation (PVSG). PVSG is related to the existing video scene graph generation (VidSGG) problem, which focuses on temporal interactions between humans and objects localized with bounding boxes in videos. However, the limitation of bounding boxes in detecting non-rigid objects and backgrounds often causes VidSGG systems to miss key details that are crucial for comprehensive video understanding. In contrast, PVSG requires nodes in scene graphs to be grounded by more precise, pixel-level segmentation masks, which facilitate holistic scene understanding. To advance research in this new area, we contribute a high-quality PVSG dataset, which consists of 400 videos (289 third-person + 111 egocentric videos) with totally 150K frames labeled with panoptic segmentation masks as well as fine, temporal scene graphs. We also provide a variety of baseline methods and share useful design practices for future work.",

author = "Jingkang Yang and Wenxuan Peng and Xiangtai Li and Zujin Guo and Liangyu Chen and Bo Li and Zheng Ma and Kaiyang Zhou and Wayne Zhang and Loy, {Chen Change} and Ziwei Liu",

year = "2023",

month = jun,

day = "17",

doi = "10.1109/CVPR52729.2023.01791",

language = "English",

isbn = "9798350301304",

series = " Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",

publisher = "IEEE",

pages = "18675--18685",

booktitle = "Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",

address = "United States",

note = "36th IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 ; Conference date: 18-06-2023 Through 22-06-2023",

url = "https://cvpr2023.thecvf.com/virtual/2023/index.html, https://openaccess.thecvf.com/CVPR2023, https://cvpr2023.thecvf.com/virtual/2023/papers.html?filter=titles, https://ieeexplore.ieee.org/xpl/conhome/10203037/proceeding",

}

Yang, J, Peng, W, Li, X, Guo, Z, Chen, L, Li, B, Ma, Z, Zhou, K, Zhang, W, Loy, CC & Liu, Z 2023, Panoptic Video Scene Graph Generation. in Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 18675-18685, 36th IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, Canada, 18/06/23. https://doi.org/10.1109/CVPR52729.2023.01791

Panoptic Video Scene Graph Generation. / Yang, Jingkang; Peng, Wenxuan; Li, Xiangtai et al.
Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023. p. 18675-18685 ( Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - Panoptic Video Scene Graph Generation

AU - Yang, Jingkang

AU - Peng, Wenxuan

AU - Li, Xiangtai

AU - Guo, Zujin

AU - Chen, Liangyu

AU - Li, Bo

AU - Ma, Zheng

AU - Zhou, Kaiyang

AU - Zhang, Wayne

AU - Loy, Chen Change

AU - Liu, Ziwei

PY - 2023/6/17

Y1 - 2023/6/17

N2 - Towards building comprehensive real-world visual perception systems, we propose and study a new problem called panoptic scene graph generation (PVSG). PVSG is related to the existing video scene graph generation (VidSGG) problem, which focuses on temporal interactions between humans and objects localized with bounding boxes in videos. However, the limitation of bounding boxes in detecting non-rigid objects and backgrounds often causes VidSGG systems to miss key details that are crucial for comprehensive video understanding. In contrast, PVSG requires nodes in scene graphs to be grounded by more precise, pixel-level segmentation masks, which facilitate holistic scene understanding. To advance research in this new area, we contribute a high-quality PVSG dataset, which consists of 400 videos (289 third-person + 111 egocentric videos) with totally 150K frames labeled with panoptic segmentation masks as well as fine, temporal scene graphs. We also provide a variety of baseline methods and share useful design practices for future work.

AB - Towards building comprehensive real-world visual perception systems, we propose and study a new problem called panoptic scene graph generation (PVSG). PVSG is related to the existing video scene graph generation (VidSGG) problem, which focuses on temporal interactions between humans and objects localized with bounding boxes in videos. However, the limitation of bounding boxes in detecting non-rigid objects and backgrounds often causes VidSGG systems to miss key details that are crucial for comprehensive video understanding. In contrast, PVSG requires nodes in scene graphs to be grounded by more precise, pixel-level segmentation masks, which facilitate holistic scene understanding. To advance research in this new area, we contribute a high-quality PVSG dataset, which consists of 400 videos (289 third-person + 111 egocentric videos) with totally 150K frames labeled with panoptic segmentation masks as well as fine, temporal scene graphs. We also provide a variety of baseline methods and share useful design practices for future work.

UR - https://cvpr2023.thecvf.com/virtual/2023/poster/23221

U2 - 10.1109/CVPR52729.2023.01791

DO - 10.1109/CVPR52729.2023.01791

M3 - Conference proceeding

SN - 9798350301304

T3 - Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

SP - 18675

EP - 18685

BT - Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

PB - IEEE

T2 - 36th IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023

Y2 - 18 June 2023 through 22 June 2023

ER -

Panoptic Video Scene Graph Generation

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this