Huh, if it's good enough, movies/TV shows dubbed with AI-clone of the original voice would be great (if we can ignore the ethics of using the actor's voice and the loss of work for the dubbing companies and actors).
Can AI detect the emotional tone of sentences yet, and recreate it in the target language?
yieldcrv 155 days ago [-]
Yes, have you used OpenAI’s voice model? It uses and reacts to tones
My favorite conversation has been getting it to tell me about marshmallow vs marshmellow spelling and pronunciation, it became very strict but patient with me
It can reply in other languages too, but I cant detect dialect as well to say
skeledrew 154 days ago [-]
In my experience, human dubbing never captures the original tone anyway. Probably never can unless it's done by people fluent in both source and target languages and're also good at voice acting. And so I have a huge preference for subs so I can appreciate the nuance in the original voices.
tomw1808 154 days ago [-]
Sometimes, very rare, the dub is actually better (yes, subjective, but still) than the original. E.g. I do find Fear and Loathing in Las Vegas in German absolutely hilarious, start to finish. Then I watched the english version - was I surprised how serious this is in original voice. It's an entirely different movie in its original tone.
bestouff 146 days ago [-]
"Wayne's World" in French is also a masterpiece. But it's been dubbed by very good people (called "Les Nuls") who understood the jokes and created appropriate ones instead of loosing them in translation when needed.
carlosjobim 154 days ago [-]
When was the last time you experienced human dubbing? They do it with amazingly high quality in some languages today. I actually feel sorry for the dubbers who now have to be actors as well, making every grunt, sigh and laugh of the people they are dubbing.
They are even dubbing reality survival shows, so somebody has to sit in a studio and groan as if they are climbing a slippery hill in Alaska.
Can't talk for the German dubbing, but the Italian version sounds natural to us Italians while the original, English version, is hard to relate to and create a bond with. The dubbing makes it "close" to home if that makes sense. You might feel it's weird because you've grown accustomed to watching the original version while also immersed in everything that sitcom portrays.
programjames 155 days ago [-]
Yes, pop in a normalizing flow that removes tone and recreates it with a small audio sample as context.
What I found is that, for cross-language use-cases, this often just applies the intonation of the “context” sample to the created sample, which, if they are from different languages, usually gives the wrong result (in the sense that it sounds off).
devindotcom 155 days ago [-]
they're workin on it
_moof 154 days ago [-]
> (if we can ignore the ethics of using the actor's voice and the loss of work for the dubbing companies and actors)
This is a shocking parenthetical.
154 days ago [-]
roenxi 154 days ago [-]
Optimistically, maybe we can replace the original actor with a synthetic voice as well. Then the issue is largely moot.
CaptainFever 154 days ago [-]
I realise that for dubbing (and voice actors in general), they have the least amount of "potential fallback legal protections" in the context of automation via generative AI:
Translating? Machine translation is already well established.
Copyright? IIRC the most popular voice generation company (ElevenLabs) uses copyright-safe models, where the sources for the base model were already consenting.
Likeness? As you said, just use a synthetic voice.
Replacement of jobs? Not really a legal issue. It's not much worse than self checkouts or driverless cars, for example. The only reason we're talking about it is because it affects white-collar workers and not blue-collar workers, and voice actors are more likely to be celebrities than cashiers, for example.
I want to learn swedish and because there are so few dubbed movies in Swedish I take the subtitles(Netflix is good at having subtitles in different languages) and text-to-speech it :)
birktj 154 days ago [-]
Why not simply watch native Swedish content instead? There should be quite a lot globally available for free from SVT.
carlosjobim 154 days ago [-]
You can only watch Nordic government channels within the Nordic countries. Unless you get a VPN for it, I guess.
sam_perez 155 days ago [-]
How would you rate the quality of the dubs?
Do you think it's usable for learning? Seems like you could end up with some quirky learnings.
woodson 155 days ago [-]
Not OP, but I agree that this could lead to questionable learning outcomes, especially since Whisper isn’t that good for low-resource languages. It’s probably fine for languages like English/Spanish/Mandarin, though.
ainonsense44 154 days ago [-]
Wenn ist das nun Stück gitt und Schlottermeyer.
sunnybeetroot 155 days ago [-]
This is fantastic! Any direction on if I wanted to change the language to something else?
vouaobrasil 155 days ago [-]
Unlike other methods of automation, AI is replacing human beings too fast. And before you say, "new jobs will be created" -- look at history. After the computer, new jobs have been created, but what kind of jobs? Every year, we are becoming more entwined in wage slavery as the wealth accumulates at the top and jobs become more meaningless.
So, no, new jobs will not be created, except the kind of jobs that crush the human spirit into oblivion so that the rich tech oligarchs can play God.
Calling something a fallacy does not mean it truly is one. I have not seen strong evidence that there is not a "lump of labor" in extremely high-rent countries like the US.
For a sufficiently small period of time, there absolutely is a "lump of labor". If I were to go into a county with robots and rent them out to employers for 1/4 of the wage of their current employees, and they all fired their employees and accepted my robots, do you really believe that all of ex-employees would be able to find work again within 6 months? Or even 70% of them? What about their new wages? Do you think these new jobs would pay as well? I have a hard time believing that to be the case.
pizza234 149 days ago [-]
There are a number of misconceptions when it comes to the lump of labor fallacy. The prevalent one is confusing the small and the big picture.
If a factory introduces automation, no doubt that the resulting job losses are a problem that must be addressed.
However, the lump of labor refers to the big picture - in the same time span, other jobs are created elsewhere. If they weren't, considering that automation started at least 200 years ago (in the most limited sense of the term), the whole planet would be out of jobs by now.
vouaobrasil 155 days ago [-]
If it is such a fallacy, then why are there fewer and fewer options for those who don't want to be entrenched in technology?
IG_Semmelweiss 154 days ago [-]
Who's upset here about the hordes of horse manure cleaners that lost their jobs due to the advent of cars? Does anyone miss the stench of a 1800s street in the summer heat?
No.
Lets ask the influencers, the twitch celebs, the podcasters id they would prefer to shovel horse crap, or play with their e-device all day.
And to be completely fair, the manufacturing jobs may be gone because elected leaders were told it was OK export the vast majority of those jobs (millions) abroad, where labor doesn't have govt benefits or red tape. This makes new jobs go to foreigners instead of staying stateside.
So, we all save a few pennies on each item made in china and sold on Amazon, at the price of nuking all US craftsmanship and artisans.
kevingadd 154 days ago [-]
This is really aggressively reductive. Job elimination isn't entirely about people who were sweeping up horse dung, and the new jobs aren't all comfy podcasting and influencer gigs.
The job elimination can mean people who spent decades honing a craft and no longer have any realistic job opportunities at a remotely equivalent pay level, and have no path into a new career because they're now decades behind (or too old to realistically train up a new skill if they're a physical laborer). Sure, it's progress, but it's weird to imply that all the labor that's being eliminated lacks value or artistry.
And the new jobs are frequently things like acting as a babysitter for an AI or sitting in a call center pressing buttons. Certainly, this is also "work", but if we're comparing it to shoveling horse manure I wouldn't necessarily consider it elevated much even if it's more comfortable.
latexr 154 days ago [-]
> Lets ask the influencers, the twitch celebs, the podcasters id they would prefer to shovel horse crap, or play with their e-device all day.
That reads like the premise of a joke: “What’s the difference between a manure cleaner and a social media influencer? The former reduces the amount of shit in your life, the latter adds to it”.
Preferences aside, if we take into account both physical and mental damage, I wonder which job is more harmful. Not only to the practitioner, but society in general. I’m not advocating for bringing back the job of horse manure cleaner, but I don’t think social media influencer should be a job either.
You made a good point regarding exporting jobs.
vouaobrasil 154 days ago [-]
> Who's upset here about the hordes of horse manure cleaners that lost their jobs due to the advent of cars?
False dichotomy. Not every modern-day twitch celeb would be shoveling manure. And another false dichotomy: I was not arguing for either NOW or THE TIME before cars. And I wonder how many people would trade their modern influence job for shoveling manure in return for living in an area with cleaner air. Also, ask the question again to people living NEXT to tire factories that stink and whose only job is collecting garbage NOW like bottles from trash because they don't have enough education to get a proper job.
Besides, I am one of those "influencers" -- not exactly on Twitch, but a full-time content creator. And I WOULD go back to those times.
Why is it that every time I bring up the dangers of technology, some techie HAS to bring up cars versus manure shovelers, as if that settles every argument about the dangers of technology? Rather intellectually stifled, I feel. And rejecting technology doesn't mean going back to the way things WERE; rather it means making changes NOW to go to a NEW future that has less technology.
Kiro 154 days ago [-]
You're setting an extremely obnoxious tone by capitalizing random words like that. I suggest that you edit your post if you want to be taken seriously.
vouaobrasil 154 days ago [-]
I would call the neighbour playing rap and full volume 'extremely obnoxious'. It's hard to believe that a few capitalized words would fall under the same category. Suggestion noted.
skeledrew 154 days ago [-]
Technology disappears when it becomes ubiquitous. It never actually becomes less, because people don't want less. I can't see for example people wanting there to be fewer lightbulbs, or even refrigerators (mainly for those in warm areas). Everybody wants safe lighting and their food unspoiled after a day (once exposed to these things), and likely won't consider the means to be "tech". And it only improves over time. AI is just another tech that's in the early stage but also on it's way to ubiquity.
vouaobrasil 154 days ago [-]
> because people don't want less.
I do, the Amish do, and quite a few other people against technology DO want less. In fact, even ordinary people (non-techies) often tell me technology is too entrenched. What about those people ditching smartphones for dumphones. I don't think that tech improves things and I DO want less.
pizza234 149 days ago [-]
The general form of what you're describing ("I want product X to have less Y", in this case, tech) has been discussed by Joel Spolsky in an article about product design.
I can't find the article, but it said something around the lines of "each Microsoft Word user thinks that there are too many functionalities, and wants less".
His remark is that each user uses different functionalities, so one can't reduce a product's functionality in a way that satisfies everybody.
Everybody would sure love to have a smartphone tailored 100% to their usage, but in real world, either they accept smartphones in toto, or they just use a brick. In generalized form, the same concept applies to tech.
skeledrew 153 days ago [-]
What do you want less of exactly? A dumbphone is still tech. Perhaps some would rather have just a land line (brings to mind that there are still places in my country without phone lines since the passage of hurricane Beryl)?
But back to the primary: is it really tech that some want less of, or the negative effects caused by its overuse/abuse? For example, IIUC the Amish are against modern tech primarily because it's changing their communities in ways they aren't fond of. But they still use tech that's not so modern, such as buggies (as opposed to just riding horses), manual plows for farming, saws and hammers for building, etc. Can anyone even go less than that? And that relative "less" only moves forward over time.
inkcapmushroom 154 days ago [-]
I would like fewer lightbulbs. Modern omnipresent light pollution is harmful to ourselves and our environment, and probably doesn't need to be at the level it is now for "safe lighting". It's okay to have some darkness sometimes, inside your house and inside your neighborhood.
skeledrew 153 days ago [-]
Agreed that some darkness is OK, particularly in unused areas of a home. And there're already solutions for that, such as motion sensors.
I doubt you want much dark in a neighborhood though, particularly if there are unsavory elements potentially roaming around. A sliver of light may be the difference between someone being attacked or a home robbed.
insane_dreamer 155 days ago [-]
The problem is that we are hurtling towards the unknown without a plan, driven by the “need” to make higher returns for shareholders and to capture the new market.
It may be that some new types of employment magically appear that soak up the jobs lost, but you can be sure there is no one working on solving that problem since the goal is to eliminate labor not create it.
Kiro 155 days ago [-]
Strange submission to post this comment on. It's not like translating and dubbing videos is the highest form of labor.
wongarsu 155 days ago [-]
It's not like it's mindless drudgery either. Losing as little as possible in translation while fitting within the constraints of subtitles (people can only read so fast) can be very challenging. And decent dubbing tries to approximately match the mouth movements, which puts further constraints on the translation and often requires creative text changes.
There is a point about industry size (not that many countries dub all their movies), but it is one of the intellectually more challenging professions.
Kiro 155 days ago [-]
Translation work is often associated with horrible working conditions and poor pay. It definitely fits OP's description of "jobs that crush the human spirit into oblivion." This might not be the case if you're a renowned translator of famous literature, but that is not representative of the majority of translators.
numpad0 154 days ago [-]
It takes a fluent, language specific bilingual with domain knowledge just to verify work. That's a high skilled job. It's just massively underappreciated because (same).
pizza234 155 days ago [-]
Voice acting (dubbing) is acting, which is a "high form of labor". In some languages/countries, most of the dubbing is performed by a few extremely good voice actors.
joe_the_user 155 days ago [-]
LLMs aren't going replace all actors or all voice actors just as they won't replace all illustrators or writers.
LLMs provide a certain level of mid/low quality content in nearly all mediums. And given that there many people producing such mid/low quality content today, LLMs will have an impact. LLMs affecting sales writers? Sure (not the best sales writers but the point is sales is mediocre but acceptable is a norm).
And say LLMs specifically. There's good evidence the technology has roughly peaked. That doesn't mean it's impact has peaked but it's indication that "all jobs at risk" might be an exaggeration.
Almondsetat 155 days ago [-]
I disagree. The actors performing the dubbing usually do an extremely poor job, and the space is infested by nepotism.
pjc50 154 days ago [-]
This is true - and it's why only a tiny, tiny fraction of content is ever translated at all.
We get both "cheaper, worse versions of material that would be dubbed anyway" but also "material that nobody would ever pay a human to dub".
ZoomerCretin 155 days ago [-]
Watch Squidgame's English dubs and tell me that couldn't be improved.
If for no other reason, this tool is a net-positive because it may raise the bar on the quality of dubs expected from dubbing actors.
victorbjorklund 154 days ago [-]
they wont replace voice actors for that. Rather it will replace voice actors for niche languages for niche content. Just like AI generated images doesnt replace artists working on the latest pixar movie but rather lets some small blog ad an image they would never had otherwise.
rvense 154 days ago [-]
But there's a lot of people who have been working as voice actors for small/medium visibility content. Toothpaste ads, Kickstarter promo videos, corporate training videos, mid-range video games... a chunk of that is going away, and it's going to hurt a bunch of people who were previously able to make a decent living off it without being a star.
notkaiho 155 days ago [-]
Have you tried? It's a genuine "skilled labour" job, both the translation and the voice acting.
42lux 155 days ago [-]
In which country? Germany and France put some effort in and it’s still not great most of the time but most languages dub only in 1 or 2 voices for all characters. Just look at the ridiculous long credits of Netflix shows.
evandrofisico 155 days ago [-]
Brazil has a long history of dubbing, not only translating but also localizing tv shows and movies. In some cases the shows are actually better dubbed, because the voice actors are better at emoting than the original cast.
blacksmith_tb 155 days ago [-]
Anime dubs from Japanese into English are often quite good, even idiomatic (I am sure there's some license taken, say for something like <<Kill La Kill>>, but the overall result is much better for it).
pizza234 155 days ago [-]
Italian voice actors are extremely good (although I personally don't like dubbing at all).
insane_dreamer 155 days ago [-]
It actually requires a fair bit of skill to do it correctly and accurately
codedokode 155 days ago [-]
It is a form of art. I hate watching dubbed videos, because the voice actors who translate it almost always are worse than original actors.
surfingdino 155 days ago [-]
There is finesse to it. You are translating from one culture to another.
fleischhauf 154 days ago [-]
the reaction is wrong. it shouldnt be "oh no, jobs are being removed" but "nice, less work more automatization, let's make sure we all benefit through less work and not only the rich with more profits"
pjc50 154 days ago [-]
> let's make sure we all benefit through less work and not only the rich with more profits
It's fairly clear by now that that is not what happens and that the real AI risk is not the "grey goo" one of everything being converted to nanomush, but the age-old one of landlordism soaking up all the returns to economic activity.
IndySun 155 days ago [-]
>...new jobs will not be created, except the kind of jobs that crush the human spirit into oblivion...
AI certainly means everyone will be able to create 'art' and as a result we'll have more art than we know what to do with, music and images are already confetti, soon so will full length 'films/movies'. That leaves anyone who can actually sing, paint, play, dance, in prime position to take up those mantles.
amonith 155 days ago [-]
What do you mean by the last sentence? Isn't it the opposite? People who could do art stuff won't be able to sustain themselves using those skills anymore because those will be devalued a lot.
IndySun 151 days ago [-]
I mean, meant to imply, live performance will become gold.
lmm 155 days ago [-]
> After the computer, new jobs have been created, but what kind of jobs? Every year, we are becoming more entwined in wage slavery as the wealth accumulates at the top and jobs become more meaningless.
What are you talking about? Many of us have tech jobs with much more comfort, creativity and autonomy than the jobs they displaced, and computerisation has made it much more practical for those who dare to strike out their own rather than needing wealthy family or friends before you can even begin to think of starting a business.
CaptainFever 154 days ago [-]
I know, right. Jobs created by computers have by far been the best paying, and most comfortable jobs that exist in human history. We're privileged.
hatenberg 154 days ago [-]
Transformer based AI is basically printing machines for the knowledge economy from existing labor patterns.
Nobody currently can say which patterns it cannot extract, hence "we always figured out new jobs" is ... challenged
SoftTalker 154 days ago [-]
> After the computer, new jobs have been created, but what kind of jobs?
Um, jobs where someone under age 30 can be earning hundreds of thousands of dollars a year programming them?
vouaobrasil 154 days ago [-]
> Um, jobs where someone under age 30 can be earning hundreds of thousands of dollars a year programming them?
Not a very fulfilling job in my opinion, except for the money.
homarp 155 days ago [-]
can't I with that creates video that have the world as potential users?
and the video can be the products, or tutoriala for another products. This allows me to do more, not less.
so maybe don't cling on to "the job" so much and hoping it somehow can fulfil your life. If the job can be automated by a machine then isn't it already meaningless and mundance and bore you to death anyway?
I agree with the point about "wealth accumulates at the top" though. Maybe Karl Marx was right about a thing or 2. Maybe the distribution of wealth should not fall into the hand of non-elected corporations. Whatever it is, it should be determined by a democratic process and not some "market mechanism" that is actually just arbitrary algorithms optimized for metrics no actual human cares about.
nickthegreek 155 days ago [-]
>AI is replacing human beings too fast.
citation needed.
RIMR 155 days ago [-]
This is a subjective statement. If you disagree with it, counter it. Asking for a citation doesn't make sense here.
chompychop 155 days ago [-]
How is it subjective? Can't terms like "replacing" and "fast" be quantified by metrics related to rates of unemployment and adoption of AI systems for tasks previously manned by humans? I'm not saying the data is readily available, but I do see a route to objectively measuring this.
SoftTalker 154 days ago [-]
It's the word "too" that makes it subjective.
bsaul 155 days ago [-]
« It’s too sweet » falls into the same measurable phenomenon, and yet remains highly subjective
numpad0 155 days ago [-]
I don't think generative AI is replacing humans at all. It's like how SQL replaced software engineers, with added bonus of copyright doubts gatekeeping common folks from exploiting it. It's obviously killing open Internet fast and encouraging power concentration too. It's worst of couples of worlds.
skummetmaelk 154 days ago [-]
Localization and dubbing is a sad endeavour. By trying to accomodate everyone's individual preference for information transmission we accomplish nothing more than reducing our ability to understand each other in the long run.
Having a Babelfish is all well and good. Until it stops working, and you realise no one can understand each other any more.
Ironically localization is often pushed by well meaning Americans who only speak one language. "Oh, you're in a French speaking region. You MUST want French language. Let me force it down your throat while I prance around virtue signalling about how inclusive we are"
pjc50 154 days ago [-]
This is a terrible take, and you should have at least included the "forced" dub disclaimer from your comment below. Without at least one of sub/dub, foreign (relative to your current location!) language content basically doesn't get consumed at all except by a very small minority of people who are very keen on the content anyway - or are speakers of the language anyway.
Now, as veterans of anime forum wars will know, subtitling is nearly always better than dubbing, and I hope this tech is capable of that as well. Most media systems let you put a whole load of subtitle tracks on and then pick one.
There's far, far too much content out there for more than a fraction of it to be ever professionally translated. While we should expect human translation review and a spot of localization for officially released works, most of the internet is just free content being given away for very little return. And that's where automatic translation is going to shine: release the non-English meme champions! Let us have a look in Bilibili!
skummetmaelk 154 days ago [-]
A big reason for the Dutch and Nordic populations being better at English than the Germans, Italians, and Spaniards, is that they were simply forced to consume media in English. They had no other option. Turns out to be an advantage.
giorgiobalduino 154 days ago [-]
As an Italian I honestly prefer dubbed versions and most Italians feel the same way. We have great dubbers whose voice we associate an actor with, so much that hearing Sylvester Stallone or Eddie Murphy speaking in his native English feels weird, almost a fraud, the breaking of a bond that's not there anymore. Perhaps I would change my view on this if I lived in USA, UK or Australia, but I don't and thus that might be the reason I can't relate with originals. Just my 2 cents.
carlosjobim 154 days ago [-]
Subtitled media in English, that is.
kevingadd 154 days ago [-]
Are you saying it's a bad thing if the creator of a work decides they want it localized or dubbed into other languages? I don't understand why you want to take that choice out of their hands.
Which languages someone speaks isn't simply a matter of "individual preference". Learning a new language takes a lot of time and energy, and people only have the time to learn a handful of languages in most cases unless they can make a career out of linguistics.
i.e. I know a sprinkling of words in various languages, and I've started learning Japanese, but I simply don't have the time to also learn Mandarin, Korean, Cantonese, etc. So I appreciate it when authors of works in those languages offer localizations into a language I can speak, or when third parties spend their time translating stuff for free to make it available to a wider audience.
What's the advantage of closing knowledge and communication off from a wider audience?
Maybe I'm misunderstanding and you're just angry about Google Translate/DeepL etc (which I have a strong distaste for since they're Fake)
skeledrew 154 days ago [-]
Nah the localization itself is fine. Where it becomes problematic is when there's no opt-out at the app level. Or perhaps it should even be opt-in.
CaptainFever 154 days ago [-]
Opt out? Do you mean you want content creators to be able to ban certain texts from being translated? That sounds like a terrible idea.
Am4TIfIsER0ppos 154 days ago [-]
No. The parent means that software and media should not only be available in French in France, for example. French produced things can be in French in France but GGP is talking about American services like google which used to always default to French for me and be resistant to changing.
But also yes a creator should be able to ban translations.
pjc50 154 days ago [-]
> software and media should not only be available in French in France
> But also yes a creator should be able to ban translations.
We've been round and round this for at least twenty years; creators like being able to ban accessibility measures like "read aloud this document" or "display it in a more readable format" or "fix the audio mix so the dialogue is audible" or "buy the DVD from a different country", but that's not exactly welfare-maximising. Are translations an accessibility measure? What about a translation into ASL?
(on the other hand, the reputational risk of a poor translation is real, and in the extreme can result in someone being cancelled for something they never even wrote)
Am4TIfIsER0ppos 154 days ago [-]
I don't mean you should be technically prevented from performing a translation, edit, or filter on something you own. I mean creators should be able to prevent the publication of a translation whether that is a different language, subtitles, or signing through the usual copyright mechanisms.
bdzr 154 days ago [-]
I'd never heard this before, but it's really interesting. This reminds me a lot of when Hollywood tried (successfully?) to bully Netflix out of adding a playback speed button. I'm not sure to what extent creators should be allowed to control how people consume their media.
skeledrew 153 days ago [-]
This is the kind of thing why I'm very anti-DRM. Once content hits my system, particularly if I've paid for it, it should be mine to do with as I please.
written-beyond 154 days ago [-]
Interesting take, never really looked at it that way.
LeoPanthera 155 days ago [-]
For years I've wanted this for live TV. Even just subtitles would be amazing. I've always wanted to be able to watch news TV from other countries.
codedokode 155 days ago [-]
As I understand, it first extracts text from original video into subtitles, translates them using external LLM, and then converts text to speech. All of this is done using thrid-party solutions, and the project seems to be just a GUI app that allows to integrate them.
You obviously cannot use this to translate songs or movies because this method loses important information like voice, intonation, etc.
So it is still better to use subtitles.
mrtksn 155 days ago [-]
Back in high school, when I got my first PC a plumber came over to fix some stuff and when he saw the computer he got excited and asked some questions and one of the questions was “how do you translate the VCD with this, I have a movie to watch but hate subtitles”.
I was like “silly dude doesn’t know how computers work” but maybe I was the silly one who can’t dare to imagine how something like that can work.
gagabity 154 days ago [-]
Cool what languages can it do?
Yandex browser does the most impressive version of this and for free but only to Russian I believe, its quite amazing it does appropriate different voices and follows the correct intonation for everyone, just takes a few seconds for a YT video.
nsonha 154 days ago [-]
This could be useful in combating fake news. In many videos especially in political news, foreign languages are dub over with sometimes nuanced translation that can skew audiences to (mis)understand the content in certain ways.
kevingadd 154 days ago [-]
A translation lacking nuance/precision (due to being the work of machine learning) can also cause significant misunderstandings, though. I'm not sure you win or lose in that regard by switching from humans to machines.
nsonha 154 days ago [-]
the errors in ML translation would be systemic and less likely to be of human arbitration, which is less dangerous. Human translation could be deliberately misleading at a specific phrase in a specific interview.
pjc50 154 days ago [-]
The AI is quite capable of inserting its own translation errors.
cyberax 155 days ago [-]
I would pay a lot for a tool that removes the freaking laugh track from videos.
I just physically can't watch them. I wanted to watch the Blackadder series, but I couldn't even get through one episode.
> Open-source tool translates and dubs videos into other languages using AI
Is there any assessment about how good the translation is ?
CyberDildonics 155 days ago [-]
Is this using an open source text to speech model or is it going out to some other internet service?
tourmalinetaco 155 days ago [-]
It can use Whisper which is open source, although optionally it can also use GoogleSpeech.
CyberDildonics 155 days ago [-]
I took a look and that seems to be speech to text / speech recognition but not text to speech.
paulkon 154 days ago [-]
Is there an open source speech-to-speech model which retains intonation, cadence and delivery?
randomgiy3142 155 days ago [-]
Because translations are copyrighted so it is complex to get legal rights for them.
ewuhic 155 days ago [-]
This one does dubbing, but is there an equivalent tool for subs?
fbnt 154 days ago [-]
Not open source, but https://fluen.ai does a good job at translating subs, while using your own or standards style guides for the target language - ie. reading speed, max characters, "chunking" sentences where it grammatically makes sense, re-adapting them etc.
cyanydeez 155 days ago [-]
So uh, what's with AI products throwing out the gold standard in testing these claims.
For example here's how weird Friends is in German: https://www.youtube.com/watch?v=nCoNSZV--z0 . Or Italian: https://www.youtube.com/watch?v=wO5qTzvyQ1s
Can AI detect the emotional tone of sentences yet, and recreate it in the target language?
My favorite conversation has been getting it to tell me about marshmallow vs marshmellow spelling and pronunciation, it became very strict but patient with me
It can reply in other languages too, but I cant detect dialect as well to say
They are even dubbing reality survival shows, so somebody has to sit in a studio and groan as if they are climbing a slippery hill in Alaska.
Can't talk for the German dubbing, but the Italian version sounds natural to us Italians while the original, English version, is hard to relate to and create a bond with. The dubbing makes it "close" to home if that makes sense. You might feel it's weird because you've grown accustomed to watching the original version while also immersed in everything that sitcom portrays.
https://arxiv.org/abs/2312.01479
This is a shocking parenthetical.
Translating? Machine translation is already well established.
Copyright? IIRC the most popular voice generation company (ElevenLabs) uses copyright-safe models, where the sources for the base model were already consenting.
Likeness? As you said, just use a synthetic voice.
Replacement of jobs? Not really a legal issue. It's not much worse than self checkouts or driverless cars, for example. The only reason we're talking about it is because it affects white-collar workers and not blue-collar workers, and voice actors are more likely to be celebrities than cashiers, for example.
https://github.com/cubbK/dubbing_ai_netflix_client
I want to learn swedish and because there are so few dubbed movies in Swedish I take the subtitles(Netflix is good at having subtitles in different languages) and text-to-speech it :)
Do you think it's usable for learning? Seems like you could end up with some quirky learnings.
So, no, new jobs will not be created, except the kind of jobs that crush the human spirit into oblivion so that the rich tech oligarchs can play God.
For a sufficiently small period of time, there absolutely is a "lump of labor". If I were to go into a county with robots and rent them out to employers for 1/4 of the wage of their current employees, and they all fired their employees and accepted my robots, do you really believe that all of ex-employees would be able to find work again within 6 months? Or even 70% of them? What about their new wages? Do you think these new jobs would pay as well? I have a hard time believing that to be the case.
If a factory introduces automation, no doubt that the resulting job losses are a problem that must be addressed.
However, the lump of labor refers to the big picture - in the same time span, other jobs are created elsewhere. If they weren't, considering that automation started at least 200 years ago (in the most limited sense of the term), the whole planet would be out of jobs by now.
No.
Lets ask the influencers, the twitch celebs, the podcasters id they would prefer to shovel horse crap, or play with their e-device all day.
And to be completely fair, the manufacturing jobs may be gone because elected leaders were told it was OK export the vast majority of those jobs (millions) abroad, where labor doesn't have govt benefits or red tape. This makes new jobs go to foreigners instead of staying stateside.
So, we all save a few pennies on each item made in china and sold on Amazon, at the price of nuking all US craftsmanship and artisans.
The job elimination can mean people who spent decades honing a craft and no longer have any realistic job opportunities at a remotely equivalent pay level, and have no path into a new career because they're now decades behind (or too old to realistically train up a new skill if they're a physical laborer). Sure, it's progress, but it's weird to imply that all the labor that's being eliminated lacks value or artistry.
And the new jobs are frequently things like acting as a babysitter for an AI or sitting in a call center pressing buttons. Certainly, this is also "work", but if we're comparing it to shoveling horse manure I wouldn't necessarily consider it elevated much even if it's more comfortable.
That reads like the premise of a joke: “What’s the difference between a manure cleaner and a social media influencer? The former reduces the amount of shit in your life, the latter adds to it”.
Preferences aside, if we take into account both physical and mental damage, I wonder which job is more harmful. Not only to the practitioner, but society in general. I’m not advocating for bringing back the job of horse manure cleaner, but I don’t think social media influencer should be a job either.
You made a good point regarding exporting jobs.
False dichotomy. Not every modern-day twitch celeb would be shoveling manure. And another false dichotomy: I was not arguing for either NOW or THE TIME before cars. And I wonder how many people would trade their modern influence job for shoveling manure in return for living in an area with cleaner air. Also, ask the question again to people living NEXT to tire factories that stink and whose only job is collecting garbage NOW like bottles from trash because they don't have enough education to get a proper job.
Besides, I am one of those "influencers" -- not exactly on Twitch, but a full-time content creator. And I WOULD go back to those times.
Why is it that every time I bring up the dangers of technology, some techie HAS to bring up cars versus manure shovelers, as if that settles every argument about the dangers of technology? Rather intellectually stifled, I feel. And rejecting technology doesn't mean going back to the way things WERE; rather it means making changes NOW to go to a NEW future that has less technology.
I do, the Amish do, and quite a few other people against technology DO want less. In fact, even ordinary people (non-techies) often tell me technology is too entrenched. What about those people ditching smartphones for dumphones. I don't think that tech improves things and I DO want less.
I can't find the article, but it said something around the lines of "each Microsoft Word user thinks that there are too many functionalities, and wants less".
His remark is that each user uses different functionalities, so one can't reduce a product's functionality in a way that satisfies everybody.
Everybody would sure love to have a smartphone tailored 100% to their usage, but in real world, either they accept smartphones in toto, or they just use a brick. In generalized form, the same concept applies to tech.
But back to the primary: is it really tech that some want less of, or the negative effects caused by its overuse/abuse? For example, IIUC the Amish are against modern tech primarily because it's changing their communities in ways they aren't fond of. But they still use tech that's not so modern, such as buggies (as opposed to just riding horses), manual plows for farming, saws and hammers for building, etc. Can anyone even go less than that? And that relative "less" only moves forward over time.
I doubt you want much dark in a neighborhood though, particularly if there are unsavory elements potentially roaming around. A sliver of light may be the difference between someone being attacked or a home robbed.
There is a point about industry size (not that many countries dub all their movies), but it is one of the intellectually more challenging professions.
LLMs provide a certain level of mid/low quality content in nearly all mediums. And given that there many people producing such mid/low quality content today, LLMs will have an impact. LLMs affecting sales writers? Sure (not the best sales writers but the point is sales is mediocre but acceptable is a norm).
And say LLMs specifically. There's good evidence the technology has roughly peaked. That doesn't mean it's impact has peaked but it's indication that "all jobs at risk" might be an exaggeration.
We get both "cheaper, worse versions of material that would be dubbed anyway" but also "material that nobody would ever pay a human to dub".
If for no other reason, this tool is a net-positive because it may raise the bar on the quality of dubs expected from dubbing actors.
It's fairly clear by now that that is not what happens and that the real AI risk is not the "grey goo" one of everything being converted to nanomush, but the age-old one of landlordism soaking up all the returns to economic activity.
AI certainly means everyone will be able to create 'art' and as a result we'll have more art than we know what to do with, music and images are already confetti, soon so will full length 'films/movies'. That leaves anyone who can actually sing, paint, play, dance, in prime position to take up those mantles.
What are you talking about? Many of us have tech jobs with much more comfort, creativity and autonomy than the jobs they displaced, and computerisation has made it much more practical for those who dare to strike out their own rather than needing wealthy family or friends before you can even begin to think of starting a business.
Nobody currently can say which patterns it cannot extract, hence "we always figured out new jobs" is ... challenged
Um, jobs where someone under age 30 can be earning hundreds of thousands of dollars a year programming them?
Not a very fulfilling job in my opinion, except for the money.
and the video can be the products, or tutoriala for another products. This allows me to do more, not less.
I agree with the point about "wealth accumulates at the top" though. Maybe Karl Marx was right about a thing or 2. Maybe the distribution of wealth should not fall into the hand of non-elected corporations. Whatever it is, it should be determined by a democratic process and not some "market mechanism" that is actually just arbitrary algorithms optimized for metrics no actual human cares about.
citation needed.
Having a Babelfish is all well and good. Until it stops working, and you realise no one can understand each other any more.
Ironically localization is often pushed by well meaning Americans who only speak one language. "Oh, you're in a French speaking region. You MUST want French language. Let me force it down your throat while I prance around virtue signalling about how inclusive we are"
Now, as veterans of anime forum wars will know, subtitling is nearly always better than dubbing, and I hope this tech is capable of that as well. Most media systems let you put a whole load of subtitle tracks on and then pick one.
There's far, far too much content out there for more than a fraction of it to be ever professionally translated. While we should expect human translation review and a spot of localization for officially released works, most of the internet is just free content being given away for very little return. And that's where automatic translation is going to shine: release the non-English meme champions! Let us have a look in Bilibili!
Which languages someone speaks isn't simply a matter of "individual preference". Learning a new language takes a lot of time and energy, and people only have the time to learn a handful of languages in most cases unless they can make a career out of linguistics.
i.e. I know a sprinkling of words in various languages, and I've started learning Japanese, but I simply don't have the time to also learn Mandarin, Korean, Cantonese, etc. So I appreciate it when authors of works in those languages offer localizations into a language I can speak, or when third parties spend their time translating stuff for free to make it available to a wider audience.
What's the advantage of closing knowledge and communication off from a wider audience?
Maybe I'm misunderstanding and you're just angry about Google Translate/DeepL etc (which I have a strong distaste for since they're Fake)
But also yes a creator should be able to ban translations.
Seems reasonable. However, given French media quotas e.g. https://www.independent.co.uk/arts-entertainment/music/news/... , companies may be providing stuff only in French to ensure that it meets a legal requirement.
> But also yes a creator should be able to ban translations.
We've been round and round this for at least twenty years; creators like being able to ban accessibility measures like "read aloud this document" or "display it in a more readable format" or "fix the audio mix so the dialogue is audible" or "buy the DVD from a different country", but that's not exactly welfare-maximising. Are translations an accessibility measure? What about a translation into ASL?
(on the other hand, the reputational risk of a poor translation is real, and in the extreme can result in someone being cancelled for something they never even wrote)
You obviously cannot use this to translate songs or movies because this method loses important information like voice, intonation, etc.
So it is still better to use subtitles.
I was like “silly dude doesn’t know how computers work” but maybe I was the silly one who can’t dare to imagine how something like that can work.
Yandex browser does the most impressive version of this and for free but only to Russian I believe, its quite amazing it does appropriate different voices and follows the correct intonation for everyone, just takes a few seconds for a YT video.
I just physically can't watch them. I wanted to watch the Blackadder series, but I couldn't even get through one episode.
If you can train an instrument model on laugh tracks Demucs should do that.
Is there any assessment about how good the translation is ?
https://github.com/jianchang512/pyvideotrans/blob/main/docs/...