VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing

Video dubbing aims to translate the original speech in a film or television program into target language, which can be achieved with cascaded system consisting of recognition, machine translation and synthesis. To ensure translated well aligned corresponding video, length/duration should as close possible that speech, requires strict length control. Previous works usually control number words characters generated by model similar source sentence, without considering isochronicity duration words/characters different languages varies. In this paper, we propose VideoDubber, tailored for task video dubbing, directly considers each token translation, match speech. Specifically, sentence guiding prediction word information, including itself how much is left remaining words. We design experiments on four language directions (German -> English, Spanish Chinese English), results show VideoDubber achieves better ability than baseline methods. make up lack real-world datasets, also construct test set collected from films provide comprehensive evaluations task.