| Paper Abstract and Keywords |
| Presentation |
2023-02-21 14:45
A Note on Improvement of Binauralization Performance Based on Multi-view Learning on 360° Videos
Masaki Yoshida, Ren Togo, Takahiro Ogawa, Miki Haseyama (Hokkaido Univ.) |
| Abstract |
(in Japanese) |
(See Japanese page) |
| (in English) |
In this paper, we propose a binaural audio generation method based on multi-view learning using 360◦ videos. Conventionally, learning visually informed binaural audio generation requires ground truth binaural audio. We generate training video data from 360◦ videos and train binaural audio generation. By using 360◦ videos, which allow users to freely manipulate their viewpoints, we can generate multiple video data with different viewing directions. Our approach enables multi-view learning based on videos of the same scene with different viewing directions. Furthermore, we conduct pre-training before binaural audio generation for learning spatial correspondence between the video frame and the audio. In the pre-training, we generate videos in which the gaze direction does not match that of the audio and predict the gap in gaze direction. By using the data generated from 360◦ videos and pre-trained networks, we can improve the accuracy of binaural audio generation. |
| Keyword |
(in Japanese) |
(See Japanese page) |
| (in English) |
Multi-modal learning / Binaural audio / 360° video / Multi-view learning / Pre-training / / / |
| Reference Info. |
ITE Tech. Rep., vol. 47, no. 6, ME2023-33, pp. 65-69, Feb. 2023. |
| Paper # |
ME2023-33 |
| Date of Issue |
2023-02-14 (MMS, ME, AIT) |
| ISSN |
Print edition: ISSN 1342-6893 Online edition: ISSN 2424-1970 |
| Download PDF |
|
| Conference Information |
| Committee |
MMS ME AIT IEICE-IE IEICE-ITS |
| Conference Date |
2023-02-21 - 2023-02-22 |
| Place (in Japanese) |
(See Japanese page) |
| Place (in English) |
Hokkaido Univ. |
| Topics (in Japanese) |
(See Japanese page) |
| Topics (in English) |
Image Processing, etc. |
| Paper Information |
| Registration To |
ME |
| Conference Code |
2023-02-MMS-ME-AIT-IE-ITS |
| Language |
Japanese |
| Title (in Japanese) |
(See Japanese page) |
| Sub Title (in Japanese) |
(See Japanese page) |
| Title (in English) |
A Note on Improvement of Binauralization Performance Based on Multi-view Learning on 360° Videos |
| Sub Title (in English) |
|
| Keyword(1) |
Multi-modal learning |
| Keyword(2) |
Binaural audio |
| Keyword(3) |
360° video |
| Keyword(4) |
Multi-view learning |
| Keyword(5) |
Pre-training |
| Keyword(6) |
|
| Keyword(7) |
|
| Keyword(8) |
|
| 1st Author's Name |
Masaki Yoshida |
| 1st Author's Affiliation |
Hokkaido University (Hokkaido Univ.) |
| 2nd Author's Name |
Ren Togo |
| 2nd Author's Affiliation |
Hokkaido University (Hokkaido Univ.) |
| 3rd Author's Name |
Takahiro Ogawa |
| 3rd Author's Affiliation |
Hokkaido University (Hokkaido Univ.) |
| 4th Author's Name |
Miki Haseyama |
| 4th Author's Affiliation |
Hokkaido University (Hokkaido Univ.) |
| 5th Author's Name |
|
| 5th Author's Affiliation |
() |
| 6th Author's Name |
|
| 6th Author's Affiliation |
() |
| 7th Author's Name |
|
| 7th Author's Affiliation |
() |
| 8th Author's Name |
|
| 8th Author's Affiliation |
() |
| 9th Author's Name |
|
| 9th Author's Affiliation |
() |
| 10th Author's Name |
|
| 10th Author's Affiliation |
() |
| 11th Author's Name |
|
| 11th Author's Affiliation |
() |
| 12th Author's Name |
|
| 12th Author's Affiliation |
() |
| 13th Author's Name |
|
| 13th Author's Affiliation |
() |
| 14th Author's Name |
|
| 14th Author's Affiliation |
() |
| 15th Author's Name |
|
| 15th Author's Affiliation |
() |
| 16th Author's Name |
|
| 16th Author's Affiliation |
() |
| 17th Author's Name |
|
| 17th Author's Affiliation |
() |
| 18th Author's Name |
|
| 18th Author's Affiliation |
() |
| 19th Author's Name |
|
| 19th Author's Affiliation |
() |
| 20th Author's Name |
|
| 20th Author's Affiliation |
() |
| 21st Author's Name |
|
| 21st Author's Affiliation |
() |
| 22nd Author's Name |
|
| 22nd Author's Affiliation |
() |
| 23rd Author's Name |
|
| 23rd Author's Affiliation |
() |
| 24th Author's Name |
|
| 24th Author's Affiliation |
() |
| 25th Author's Name |
|
| 25th Author's Affiliation |
() |
| 26th Author's Name |
/ / |
| 26th Author's Affiliation |
()
() |
| 27th Author's Name |
/ / |
| 27th Author's Affiliation |
()
() |
| 28th Author's Name |
/ / |
| 28th Author's Affiliation |
()
() |
| 29th Author's Name |
/ / |
| 29th Author's Affiliation |
()
() |
| 30th Author's Name |
/ / |
| 30th Author's Affiliation |
()
() |
| 31st Author's Name |
/ / |
| 31st Author's Affiliation |
()
() |
| 32nd Author's Name |
/ / |
| 32nd Author's Affiliation |
()
() |
| 33rd Author's Name |
/ / |
| 33rd Author's Affiliation |
()
() |
| 34th Author's Name |
/ / |
| 34th Author's Affiliation |
()
() |
| 35th Author's Name |
/ / |
| 35th Author's Affiliation |
()
() |
| 36th Author's Name |
/ / |
| 36th Author's Affiliation |
()
() |
| Speaker |
Author-1 |
| Date Time |
2023-02-21 14:45:00 |
| Presentation Time |
15 minutes |
| Registration for |
ME |
| Paper # |
MMS2023-13, ME2023-33, AIT2023-13 |
| Volume (vol) |
vol.47 |
| Number (no) |
no.6 |
| Page |
pp.65-69 |
| #Pages |
5 |
| Date of Issue |
2023-02-14 (MMS, ME, AIT) |