Paper Abstract and Keywords |
Presentation |
2023-02-21 14:45
A Note on Improvement of Binauralization Performance Based on Multi-view Learning on 360° Videos Masaki Yoshida, Ren Togo, Takahiro Ogawa, Miki Haseyama (Hokkaido Univ.) |
Abstract |
(in Japanese) |
(See Japanese page) |
(in English) |
In this paper, we propose a binaural audio generation method based on multi-view learning using 360◦ videos. Conventionally, learning visually informed binaural audio generation requires ground truth binaural audio. We generate training video data from 360◦ videos and train binaural audio generation. By using 360◦ videos, which allow users to freely manipulate their viewpoints, we can generate multiple video data with different viewing directions. Our approach enables multi-view learning based on videos of the same scene with different viewing directions. Furthermore, we conduct pre-training before binaural audio generation for learning spatial correspondence between the video frame and the audio. In the pre-training, we generate videos in which the gaze direction does not match that of the audio and predict the gap in gaze direction. By using the data generated from 360◦ videos and pre-trained networks, we can improve the accuracy of binaural audio generation. |
Keyword |
(in Japanese) |
(See Japanese page) |
(in English) |
Multi-modal learning / Binaural audio / 360° video / Multi-view learning / Pre-training / / / |
Reference Info. |
ITE Tech. Rep., vol. 47, no. 6, ME2023-33, pp. 65-69, Feb. 2023. |
Paper # |
ME2023-33 |
Date of Issue |
2023-02-14 (MMS, ME, AIT) |
ISSN |
Print edition: ISSN 1342-6893 Online edition: ISSN 2424-1970 |
Download PDF |
|
Conference Information |
Committee |
MMS ME AIT IEICE-IE IEICE-ITS |
Conference Date |
2023-02-21 - 2023-02-22 |
Place (in Japanese) |
(See Japanese page) |
Place (in English) |
Hokkaido Univ. |
Topics (in Japanese) |
(See Japanese page) |
Topics (in English) |
Image Processing, etc. |
Paper Information |
Registration To |
ME |
Conference Code |
2023-02-MMS-ME-AIT-IE-ITS |
Language |
Japanese |
Title (in Japanese) |
(See Japanese page) |
Sub Title (in Japanese) |
(See Japanese page) |
Title (in English) |
A Note on Improvement of Binauralization Performance Based on Multi-view Learning on 360° Videos |
Sub Title (in English) |
|
Keyword(1) |
Multi-modal learning |
Keyword(2) |
Binaural audio |
Keyword(3) |
360° video |
Keyword(4) |
Multi-view learning |
Keyword(5) |
Pre-training |
Keyword(6) |
|
Keyword(7) |
|
Keyword(8) |
|
1st Author's Name |
Masaki Yoshida |
1st Author's Affiliation |
Hokkaido University (Hokkaido Univ.) |
2nd Author's Name |
Ren Togo |
2nd Author's Affiliation |
Hokkaido University (Hokkaido Univ.) |
3rd Author's Name |
Takahiro Ogawa |
3rd Author's Affiliation |
Hokkaido University (Hokkaido Univ.) |
4th Author's Name |
Miki Haseyama |
4th Author's Affiliation |
Hokkaido University (Hokkaido Univ.) |
5th Author's Name |
|
5th Author's Affiliation |
() |
6th Author's Name |
|
6th Author's Affiliation |
() |
7th Author's Name |
|
7th Author's Affiliation |
() |
8th Author's Name |
|
8th Author's Affiliation |
() |
9th Author's Name |
|
9th Author's Affiliation |
() |
10th Author's Name |
|
10th Author's Affiliation |
() |
11th Author's Name |
|
11th Author's Affiliation |
() |
12th Author's Name |
|
12th Author's Affiliation |
() |
13th Author's Name |
|
13th Author's Affiliation |
() |
14th Author's Name |
|
14th Author's Affiliation |
() |
15th Author's Name |
|
15th Author's Affiliation |
() |
16th Author's Name |
|
16th Author's Affiliation |
() |
17th Author's Name |
|
17th Author's Affiliation |
() |
18th Author's Name |
|
18th Author's Affiliation |
() |
19th Author's Name |
|
19th Author's Affiliation |
() |
20th Author's Name |
|
20th Author's Affiliation |
() |
21st Author's Name |
|
21st Author's Affiliation |
() |
22nd Author's Name |
|
22nd Author's Affiliation |
() |
23rd Author's Name |
|
23rd Author's Affiliation |
() |
24th Author's Name |
|
24th Author's Affiliation |
() |
25th Author's Name |
|
25th Author's Affiliation |
() |
26th Author's Name |
/ / |
26th Author's Affiliation |
()
() |
27th Author's Name |
/ / |
27th Author's Affiliation |
()
() |
28th Author's Name |
/ / |
28th Author's Affiliation |
()
() |
29th Author's Name |
/ / |
29th Author's Affiliation |
()
() |
30th Author's Name |
/ / |
30th Author's Affiliation |
()
() |
31st Author's Name |
/ / |
31st Author's Affiliation |
()
() |
32nd Author's Name |
/ / |
32nd Author's Affiliation |
()
() |
33rd Author's Name |
/ / |
33rd Author's Affiliation |
()
() |
34th Author's Name |
/ / |
34th Author's Affiliation |
()
() |
35th Author's Name |
/ / |
35th Author's Affiliation |
()
() |
36th Author's Name |
/ / |
36th Author's Affiliation |
()
() |
Speaker |
Author-1 |
Date Time |
2023-02-21 14:45:00 |
Presentation Time |
15 minutes |
Registration for |
ME |
Paper # |
MMS2023-13, ME2023-33, AIT2023-13 |
Volume (vol) |
vol.47 |
Number (no) |
no.6 |
Page |
pp.65-69 |
#Pages |
5 |
Date of Issue |
2023-02-14 (MMS, ME, AIT) |