Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study

Wenbai Xue; Cheng Hung Lo

doi:10.1007/978-3-031-06047-2_39

Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study

Wenbai Xue, Cheng Hung Lo^*

^*Corresponding author for this work

Xi'an Jiaotong-Liverpool University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

When watching films made and displayed with Virtual Reality approaches, the viewers can freely move the visual field, resulting in possible disruptions in the narratives designed by the directors. This phenomenon demands novel narrative strategies in Cinematic Virtual Reality (CVR) to effectively guide a viewer’s attention in following important plots. In this study, we evaluate the effect of using sound as a guiding mechanism in CVR. We conduct experiments to analyze the participants’ responses to the sound cues outside the field of view with the eye-tracking technique. Statistical methods are then used to infer the significance of the differences among the responses. The experiments are conducted in a virtual scene with low complexity, reducing the possible confounding effects of the variety of visual elements. The results show that the viewer's visual attention can be guided by sounds sourced at the range outside the field of view. More specifically, the viewers react significantly better to sound stimuli varying in horizontal directions than those in vertical directions. Furthermore, different types of sounds also significantly affect the viewers’ attention in the virtual scene.

Original language	English
Title of host publication	Cross-Cultural Design. Applications in Learning, Arts, Cultural Heritage, Creative Industries, and Virtual Reality - 14th International Conference, CCD 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Proceedings
Editors	Pei-Luen Patrick Rau
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	520-535
Number of pages	16
ISBN (Print)	9783031060465
DOIs	https://doi.org/10.1007/978-3-031-06047-2_39
State	Published - 2022
Externally published	Yes
Event	14th International Conference on Cross-Cultural Design, CCD 2022 Held as Part of the 24th HCI International Conference, HCII 2022 - Virtual, Online Duration: 26 06 2022 → 01 07 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13312 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	14th International Conference on Cross-Cultural Design, CCD 2022 Held as Part of the 24th HCI International Conference, HCII 2022
City	Virtual, Online
Period	26/06/22 → 01/07/22

Bibliographical note

Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Keywords

3D sound
Cinematic virtual reality
Eye-tracking
Sound-guided framing
Visual attention

Access to Document

10.1007/978-3-031-06047-2_39

Cite this

Xue, W., & Lo, C. H. (2022). Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study. In P.-L. P. Rau (Ed.), Cross-Cultural Design. Applications in Learning, Arts, Cultural Heritage, Creative Industries, and Virtual Reality - 14th International Conference, CCD 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Proceedings (pp. 520-535). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13312 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-06047-2_39

Xue, Wenbai ; Lo, Cheng Hung. / Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study. Cross-Cultural Design. Applications in Learning, Arts, Cultural Heritage, Creative Industries, and Virtual Reality - 14th International Conference, CCD 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Proceedings. editor / Pei-Luen Patrick Rau. Springer Science and Business Media Deutschland GmbH, 2022. pp. 520-535 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{4ce6f293e83d4b159bf0faa03df87472,

title = "Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study",

abstract = "When watching films made and displayed with Virtual Reality approaches, the viewers can freely move the visual field, resulting in possible disruptions in the narratives designed by the directors. This phenomenon demands novel narrative strategies in Cinematic Virtual Reality (CVR) to effectively guide a viewer{\textquoteright}s attention in following important plots. In this study, we evaluate the effect of using sound as a guiding mechanism in CVR. We conduct experiments to analyze the participants{\textquoteright} responses to the sound cues outside the field of view with the eye-tracking technique. Statistical methods are then used to infer the significance of the differences among the responses. The experiments are conducted in a virtual scene with low complexity, reducing the possible confounding effects of the variety of visual elements. The results show that the viewer's visual attention can be guided by sounds sourced at the range outside the field of view. More specifically, the viewers react significantly better to sound stimuli varying in horizontal directions than those in vertical directions. Furthermore, different types of sounds also significantly affect the viewers{\textquoteright} attention in the virtual scene.",

keywords = "3D sound, Cinematic virtual reality, Eye-tracking, Sound-guided framing, Visual attention",

author = "Wenbai Xue and Lo, {Cheng Hung}",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 14th International Conference on Cross-Cultural Design, CCD 2022 Held as Part of the 24th HCI International Conference, HCII 2022 ; Conference date: 26-06-2022 Through 01-07-2022",

year = "2022",

doi = "10.1007/978-3-031-06047-2_39",

language = "英语",

isbn = "9783031060465",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "520--535",

editor = "Rau, {Pei-Luen Patrick}",

booktitle = "Cross-Cultural Design. Applications in Learning, Arts, Cultural Heritage, Creative Industries, and Virtual Reality - 14th International Conference, CCD 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Proceedings",

address = "德国",

}

Xue, W & Lo, CH 2022, Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study. in P-LP Rau (ed.), Cross-Cultural Design. Applications in Learning, Arts, Cultural Heritage, Creative Industries, and Virtual Reality - 14th International Conference, CCD 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13312 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 520-535, 14th International Conference on Cross-Cultural Design, CCD 2022 Held as Part of the 24th HCI International Conference, HCII 2022, Virtual, Online, 26/06/22. https://doi.org/10.1007/978-3-031-06047-2_39

Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study. / Xue, Wenbai; Lo, Cheng Hung.
Cross-Cultural Design. Applications in Learning, Arts, Cultural Heritage, Creative Industries, and Virtual Reality - 14th International Conference, CCD 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Proceedings. ed. / Pei-Luen Patrick Rau. Springer Science and Business Media Deutschland GmbH, 2022. p. 520-535 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13312 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study

AU - Xue, Wenbai

AU - Lo, Cheng Hung

PY - 2022

Y1 - 2022

N2 - When watching films made and displayed with Virtual Reality approaches, the viewers can freely move the visual field, resulting in possible disruptions in the narratives designed by the directors. This phenomenon demands novel narrative strategies in Cinematic Virtual Reality (CVR) to effectively guide a viewer’s attention in following important plots. In this study, we evaluate the effect of using sound as a guiding mechanism in CVR. We conduct experiments to analyze the participants’ responses to the sound cues outside the field of view with the eye-tracking technique. Statistical methods are then used to infer the significance of the differences among the responses. The experiments are conducted in a virtual scene with low complexity, reducing the possible confounding effects of the variety of visual elements. The results show that the viewer's visual attention can be guided by sounds sourced at the range outside the field of view. More specifically, the viewers react significantly better to sound stimuli varying in horizontal directions than those in vertical directions. Furthermore, different types of sounds also significantly affect the viewers’ attention in the virtual scene.

AB - When watching films made and displayed with Virtual Reality approaches, the viewers can freely move the visual field, resulting in possible disruptions in the narratives designed by the directors. This phenomenon demands novel narrative strategies in Cinematic Virtual Reality (CVR) to effectively guide a viewer’s attention in following important plots. In this study, we evaluate the effect of using sound as a guiding mechanism in CVR. We conduct experiments to analyze the participants’ responses to the sound cues outside the field of view with the eye-tracking technique. Statistical methods are then used to infer the significance of the differences among the responses. The experiments are conducted in a virtual scene with low complexity, reducing the possible confounding effects of the variety of visual elements. The results show that the viewer's visual attention can be guided by sounds sourced at the range outside the field of view. More specifically, the viewers react significantly better to sound stimuli varying in horizontal directions than those in vertical directions. Furthermore, different types of sounds also significantly affect the viewers’ attention in the virtual scene.

KW - 3D sound

KW - Cinematic virtual reality

KW - Eye-tracking

KW - Sound-guided framing

KW - Visual attention

UR - http://www.scopus.com/inward/record.url?scp=85133163984&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-06047-2_39

DO - 10.1007/978-3-031-06047-2_39

M3 - 会议稿件

AN - SCOPUS:85133163984

SN - 9783031060465

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 520

EP - 535

BT - Cross-Cultural Design. Applications in Learning, Arts, Cultural Heritage, Creative Industries, and Virtual Reality - 14th International Conference, CCD 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Proceedings

A2 - Rau, Pei-Luen Patrick

PB - Springer Science and Business Media Deutschland GmbH

T2 - 14th International Conference on Cross-Cultural Design, CCD 2022 Held as Part of the 24th HCI International Conference, HCII 2022

Y2 - 26 June 2022 through 1 July 2022

ER -

Xue W, Lo CH. Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study. In Rau PLP, editor, Cross-Cultural Design. Applications in Learning, Arts, Cultural Heritage, Creative Industries, and Virtual Reality - 14th International Conference, CCD 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 520-535. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-06047-2_39

Sound-Guided Framing in Cinematic Virtual Reality – an Eye-Tracking Study

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this