How to translate videos into different languages

Hola! Ciao! Olá! Hallo! Witam!

AI has officially turned me into Mr. Worldwide.

If you missed the video I posted on social media earlier this week, your mind is about to be blown: TikTok | Instagram | YouTube

I made this video after seeing the viral clip of Lionel Messi speaking English. I’ve been watching Messi since I was 8 years old and always had to rely on subtitles or translators speaking over him to understand what he was saying. Hearing him speak English in his own voice scratched my brain in the most satisfying way possible.

I was impressed by how realistic and natural it looked when I tried it myself. Native speakers told me in the comments that the translations were very good, and even “perfect” at times. But I should apologize to the people of Brazil for not using their flag on the Portuguese part as it was their dialect. Desculpe!

How you can try it out

Making this didn’t require any extreme editing and was much easier than I thought. I used HeyGen’s Video Translate tool to make it happen.

Step 1: Go to HeyGen’s Video Translate tool and make an account

When I created my account, I got 2 free credits. Each credit lets you translate a one-minute video. So you can try this out without getting your wallet out.

Step 2: Upload a video and select the target language

They only allow videos up to 5 minutes long. Once you’ve uploaded your video, select the target language you want to translate it to.

Available languages (as of September 21): English, Spanish, French, Italian, Hindi, German, Polish, Portuguese, Mandarin, Japanese, Dutch, Turkish, and Korean.

Step 3: Just wait

Once you submit your video, all you have to do is wait. When I tried my first video, it took a few hours for it to be complete. The processing time is faster if you upgrade to a paid plan. Either way, HeyGen will send you an email once the translation is complete.

What this means for the future of content and media 🔮

This type of technology has applications far and wide. If you can imagine all the types of video content we consume or interact with, you start to realize the massive potential:

• Movies
• TV Shows
• YouTube videos
• Online courses
• Interviews
• Livestreams

The main benefit that binds them all is the power to lower language barriers and make content more accessible to a wider audience. There’s an abundance of insightful, entertaining, and heartwarming video content that we don’t consume because it’s not in our language. We can go beyond our content “bubble” and feel more immersed in foreign content with new perspectives and distinctive creative expressions.

An important question naturally arises: “What will happen to voice actors? Does this technology make them obsolete?”

It’s important to remember that I’ve only recorded a few seconds of myself talking in each language, and it would be premature to extrapolate this into a death sentence for the voice acting industry.

I believe this technology will impact people in different ways:

1. Helping small- and medium-sized creators lower language barriers at an affordable price

Today, not everyone who makes video content has the resources to hire voice actors and translate their movies, shows, or videos into multiple languages. At most, they create subtitles and that’s it. A prime example of this is the countless YouTube channels that are only available in one language.

The biggest channels, like MrBeast who has dubbed his videos in 11 different languages, can afford voice actors. But he’s an outlier in the ocean of small- to medium-sized YouTube channels run by a few people with limited resources.

These channels were never going to afford or invest time in hiring and recording voice actors for each language. But these new AI tools can open up a new door to dub videos at an affordable price. By lowering language barriers, they can now tap into an international audience and grow their channel/business to (maybe) reach MrBeast-level success.

Yesterday, YouTube announced that they’re bringing an AI-dubbing tool to creators along with other AI-powered tools. Read their announcement

2. The impact on Hollywood-tier movies and TV shows

Of course, we still need to talk about the impact on voice actors who are regularly hired to dub blockbuster movies and TV shows. Will they stop hiring voice actors and just replace them with AI tools?

If you’ve been reading Year 2049 for a while, you know that I don’t like blanket statements and generalizations because reality is complex and nuanced.

Those who do replace voice actors with AI tools might be disappointed, and even alienate some native speakers. Let’s take a look at some of the more critical comments on my video:

The translations are good but they seem to miss the mark in sounding natural. Some friends also told me that they felt like a machine was speaking. Maybe this will improve over time, but there’s still a need for human translators to use the right words and convey the right meaning and emotions when moving between languages.

If you asked me what the best solution to improving dubbed video content, I would say this:

• Use human translators to translate the content because they can use words, expressions, and dialogue that sound natural to a native speaker
• Use human voice actors that speak the language to record the audio because their accent sounds natural while maintaining the emotion/tone of the original video
• Use AI to regenerate the video and map the dubbed voiceover to the original actor’s lips to make it seem more natural. The biggest complaint (probably) about dubbed videos is the mismatch between the audio and an actor’s lip movements.

If we can map a native speaker’s voice to the lips of the original actor, that would be the highest-quality output with this type of technology.

Like I’ve said multiple times before, humans and AI work better together than either of them alone.

Someone asking the smart questions: Terry

I couldn’t end this post without including Terry’s insightful question… 🤣

‍