[TMP-081] Multi-layer evaluation sets for speech translation of web-based meetings
The evaluation and development data sets for speech translation for meetings
Within the scope of the project, we have created evaluation and development data sets for speech translation for meetings (English->Latvian, Latvian->English, and Lithuanian- >English) (http://hdl.handle.net/20.500.12574/74),. We contributed speech translation evaluation and development data sets for English->Latvian (4 hours and 40 minutes), Latvian->English (4 hours and 52 minutes), and Lithuanian->English (3 hours and 31 minutes). Additionally, we created an automatic minuting test set for the AutoMin 2023 shared task on automatic creation of meeting summaries (“minutes”) for English and Czech (https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-4692). We created for Task A: 10 English + 10 Czech meeting transcripts with manually created reference minutes and for Task D: manually created alignments between AutoMin 2021 system outputs and transcripts. We contributed to the human evaluation of machine translation systems for the Seventh Conference on Machine Translation for the English->Czech translation direction. The microproject produced data that will be beneficial for future developments of speech translation and automatic minuting systems. The data will allow us to better understand capabilities of these systems and identify potential areas of improvement. The results will contribute towards developing robust and trustworthy AI systems. Furthermore, the project contributed towards mobilization of the research landscape by involving Charles University and an industry partner (Tilde).
We show that the proposed approach provides high-quality semantic segmentation from the robot’s perspective, with accuracy comparable to the original one. In addition, we exploited the gained information and improved the recognition performance of the deep network for the lower viewpoints and showed that the small robot alone is capable of generating high-quality semantic maps for the human partner. The computations are close to real time, so the approach enables interactive applications.
Tangible Outcomes
- evaluation and development data sets for speech translation for meetings (for English->Latvian, Latvian->English, and Lithuanian->English) (http://hdl.handle.net/20.500.12574/74 )
- ELITIR minuting cortpus: an automatic minuting test set for the AutoMin 2023 shared task on automatic creation of meeting summaries (“minutes”) for English and Czech (https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-4692 )
Partners
- Tilde
- Charles University