Captions "Temperature" setting to shorten sentences for better readability

[Jarle Leirpoll]

The Norwegian AI trancription/captioning service "Randi" has introduced "temperature" setting for captions.
I want this in Premiere Pro! It would be great for universal design.

Here's what they say: 

Choose “temperature” and preferred sentence length when transcoding your video to get the sentence length that best suits your particular video. This will help you shorten the sentences in the subtitles so that they better suit your preferred style. At high temperatures (50-70%), Randi (the AI) will also try to make the sentences more “general” and readable, as we know from TV for example.

[R Neil Haugen]

Oh yes, please! That sounds awesome ...

[mattchristensen]

@Jarle Leirpoll is this "temperature" only about deciding where to split the transcript into caption segments? Or is it also changing the actual words?

[Jarle Leirpoll]

It is changing the words. Shortening sentences and making them clearer. Repeated and unneccesary words are removed. It's like giving ChatGTP a 300 word document and asking it to make it 200 words.

[Stan Jones]

A fascinating set of issues. Searching "AI temperature setting" gives this https://gpt.space/blog/how-to-use-openai-model-temperature-for-better-ai-chat-responses:

Essentially, a higher temperature setting allows for more randomness in the model's output, while a lower temperature produces more predictable and consistent responses. For transformation tasks like data extraction or grammar fixes, a low temperature of 0 to 0.3 is often ideal. However, for writing tasks where you want more creative and varied responses, a higher temperature closer to 0.5 is usually better.

So the usual goal of transcribing video is zero temperature to produce as accurate a transcript as possible. Accessibility standards might even require this. But when creating text on screen that is to be descriptive/informative, the AI assisted "writing" task/universal design concepts open a new frontier.

The PR transcript gives the words and punctuation for the captions it produces. But the caption length etc is determined by the caption creation dialogue/parameters.

Stan

[cdavid1760]

great idea

[Bretacious2]

Jarle
Has anyone done a survey of the target audience? 
Those are the opinions we need.
Hard of hearing and deaf people are likely the biggest audience needing captions.
I promise you they will have strong feelings about this, but I leave it to them to explain.

[R Neil Haugen]

We don't have any hearing issues, but we use a lot of captions when watching, especially if they're from the other side of the pond speaking something sort of like English, whatever it is they do speak over there.

Or just we don't want to bother others around us while watching something. As do others we know. So there's a lot of "content users" that use captions even over the audio much of the time.

And there's those that can't hear, or hear only poorly too ... so there's a high percentage of captions use these days.

[Jarle Leirpoll]

@Bretacious2  
Captions are used EVERYWHERE these days. It's required (as in: illegal not to have them) for most big content providers.

• For people with reading difficulties, shorter sentences (fewer words) makes it easier to follow the story.
• For hearing impaired, shorter sentences means they can focus more on the visuals, hence following the story better.
• For translation subtitles, too many words makes it hard to read, as you're also supposed to be watching the content.
• Same goes for quick SoMe stories and reels, fewer words makes it easier for everyone who turn audio off to follow the visual story. 

In almost every case, fewer words means a clearer story. It's also better for storytellers. If I edit a piece, use a lot of time on color grading, nice graphis, etc. - then why would I like the audience to spend most of their time reading captions? It makes no sense. Short sentences makes for a better viewing experience.

Of course, if it's a scientific presentation where you need absolute accuracy for citations etc. you would not use this feature.

[Bretacious2]

Hi Jarle
Besides my decades as an editor, I was also a sign language interpreter for
the deaf. I learned a lot about what it means to present content to them.
Unfortunately what I'm seeing yet again is hearing people making decisions
about what is best for the deaf/hh.

I asked what if any polling has been done of those people, not what you
think is best for them.

I don't claim to know either.

But we need to ask the intended audience what they prefer.

[R Neil Haugen]

What you're missing is the deaf aren't the only intended audience. 

[Bretacious2]

I'm not missing that there are various audiences who use captions.
However Captions were actually invented for the deaf. They have expanded use over the years, but the audience who needs them all the time are deaf viewers. If not for their enthusiastic use, back when caption decoders cost $800, they wouldn't be ubiqutous now.
I ask why don't we do a survery of ALL caption users and see what THEY would like, rather than tell them what they need. 

[R Neil Haugen]

That last was not clear in your post, however. You only mentioned one user-group. "We" can't read minds on forums, which is why clarity of comments are so important.

In general, I also prefer wide-based checking on things. As for instance, UI design. There are several different major 'schools' of UI design, which vary dramatically on some thing. So one "expert" group insists that X is how the general user thinks/sees/works/whatever, but another also notable expert group/org insists that Z is the correct "human" response.

There's been some intriguing discussions here on that ... where someone thinks one UI design org/whatever is "the" only real expert on UI design. And other cite 2-3 other as-notable UI design orgs.

At some point, well, how about asking people, right?   😉

[Bretacious2]

Neil
This is where it takes me when I reply to you.
If it's going to the wrong group or no group, excuse my unfamiliarity with posting here. Please tell me how to correct it.
My basic point is captioning is a service. It makes sense to poll those using the service to see what they like/don't like and need.
I'm a user, not developer. I caption programs. 
However I'm not deaf, hard of hearing or rely on captions for my own use.
Subtitles are a whole other kettle of fish IMO since they involve going from one language to another, rather than one form of a language to a different form of the same language. 
I think developers would benefit from the feedback of many actual users. 
You can contact the NAD [National Association of the Deaf], Gallaudet University and CSUN [Cal State University Northridge] Center on Deafness. These are the premiere organizations that I think would be happy to provide useful info.

[Jarle Leirpoll]

This discussion is a complete sidetrack. We don't need to discuss why or if one should do this. Please stop that, and focus on how this could be implemented.

Professional translators, interpretors and subtitle writes do this all the time. It's what their profession is about. They're not parrots, repeating every word that is said. It's also what the Randi AI team have been developing this for: To get results more like what a pro would give you.

So it's not about "should we or should we not". It's about should we have to do it manually or not.

All I'm asking is that we get help from AI to do what the professionals have been doing for decades. 

If you don't like it, don't use it. If you think we shouldn't get this feature, then don't vote for it. If you have input about HOW this should be implemented, or how the UI would best be designed, etc. — then please let us know.

[Bretacious2]

Neil
I've been involved since captions started. Many deaf still have problems with understanding them.
It's hardly a side issue, since they are your audience.
Ignore me if I bore you.

[R Neil Haugen]

You still seem to misunderstand. I have never said nor implied the deaf were not important. Not whatsoever.

But at the same time, they are not the only audience for captions.

For instance, like many, we are watching the French TV show Astrid. And as none of my family speak French what so ever, though I've had to sing it in vocal competitions in the past ... we totally rely on the captions to know what's going on.

So, yes, the deaf are important, as are all other users.

And in college, I was heavily around those working on ASL, by the by. Been around this many years. That was what ... 50 years ago now?

[Bretacious2]

Neil
"That was what ... 50 years ago now?" Deaf people have been around ever since people existed. Not sure your point.
However they are the single group who cannot intuit or parse spoken language. We can listen to French, Spanish, etc and understand a certain amount. 
But I keep feeling this pushback as if deaf people aren't that important anymore which is the ultimate irony about captions.
I simply asked if anyone surveyed ALL those who use captions to see what people want to see, not what we want to enforce on them.
The pushback tells me I'm wasting my time even asking the question.

[R Neil Haugen]

And it's puzzling to me.

But I don't know how i can be clearer. 

I have never said one word against getting deaf people involved, surveyed, whatever. Period. How you read that in is simply beyond me. And I am saddened you keep repeating that. It isn't true in any sense.

Of course deaf people should be considered! But I said that I thought rather clearly in my first post.

I have simply stated that there are also other valid real people who need transcription in the media.

Though I do thoroughly disagree "we" all know enough of any other language to get by. I know a number of other friends watching Astrid, to continue that example.

One friend spends weeks a year in Paris, and is fairly fluent in conversational French, reads and speaks it pretty well,  along with German  Norwegian, Italian, Spanish, and one dialect of Chinese.

He can't make out one word in ten of the French spoken as-is in Astrid, he says. So even for him, transcripts are required to have any idea what is going on.

How you nake the assumption that I should be able to understand enough of the audio in Astrid to gave any clue is utterly beyond me.

Please , do keep participating here! We need as many people of the user base commenting as possible, on all sorts of things.

But I ask you to understand every one of us is different, does and sees everything differently from everyone else. Which naturally means that even those who agree with you in the main will have other thoughts and ideas.

I prefer the aisle conversations at NAB, with five people every one of which thinjs differently from the others. It is a fascinating way to learn.

[Bretacious2]

I think we all share the same goal of easing and expanding communication.
It's great you have multi-lingual friends. I also speak/sign several and
it's enjoyable to converse with people in their own languages.

The main difference I've seen over several decades is hearing people can
often find other ways to communicate, using different languages like you
say.

Deaf people's world is circumscribed; they have sign language, writing,
lipreading [one of the worse communication methods] and captions.
That's it.
Every new person learning sign, is someone they likely wouldn't know
otherwise or at best smile at. That's how their world expands.
The only point I thought I was making is why not do a survey of *all *caption
users?
If you folks can create SDH, then that had to rely on some kind of research
I would hope.

I didn't invent any of this, but I learned to sign, translate to/from
French as well as create captions and subtitles with the tools you folks
invented.
I'm not a tech, but a user. I just want to keep deaf/hh on the same playing
field we all enjoy.

[R Neil Haugen]

First, to be clear, I'm another user. Never been an employee of Adobe or any other company really. Owned my own business all my adult life.

And yes, the college I went to was, at that time, one of the best 'teacher education' colleges in the country, and had a large presence of sign language teaching. Many people I knew were taking those courses. For a while I even knew a few things, but I haven't used that in so long I've forgotten them.

And of course, I've had deaf acquaintances, and severely hearing impaired but not quite deaf friends. So I've been around "them" as just other people around me. All of us trying to figure out how this whole communication thing works, no matter the level of hearing.

Plus ... for a while I spoke fairly decent conversational German, though again, not using it for 30+ years does put a dent in it. Some words and phrases are still there, but not much.

I can sort of get by på Norsk, at need, but not nearly as well as I did 30 years ago.

French ... I took classical voice lessons a few years ago, and naturally singing in Italian is absolutely a must. German and French highly advised also. The Italian I did fine in, the German was a no-brainer, French ... ah, French ...

I did sing in one vocal competition where at least one art song/aria in French was a required thing, and ... uff da. I worked very hard at it ... my vocal teacher had grown up bilingual English (American that is) and French ... her mom being native in that language ... and my daughter took to French in high school amazingly, and really mastered the sounds and the word flow and all. She was home for the summer and helped also.

But I simply do not hear French 'sounds' I guess, not just "well", but pretty much at all. And I could listen to others speak and sing French, and could not internally hear the difference between my sound and theirs.

Which I did easily in Italian, German, and Norwegian. And ... while all three judges agreed my French was terrible, two of them felt my Italian, German, and English pieces were well enough to award me something. The other judge absolutely refused as he clearly felt my French simply unacceptable as even an attempt.

I had spent a couple hours a day for months working on that French alone. Sigh.

For the show Astrid, I can't even make out a single sound, let alone, combine them to make words. It's just mumbling to me. Even when they only say one word, and 'merde' is in the transcription, I couldn't have told you that was what they said from listening.

We humans are all different, every one of us is unique, period. In listening to aisle-way discusions about editing practices at NAB every year, you have any five editors, all will do nearly everything different from all of the others in that conversation. No one works alike, even in the same office.

But then, as an autistic person, I'm perhaps more aware of the unique-ness of every individual than many of my NT peers. (Neuro-typical, or alleletic ... those supposedly "normal" people ... )

[Bretacious2]

Thanks for telling me more about your experiences.
I taught autistic as well as deaf children, learning disabled and
deaf-blind.
I think it greatly informed me about the world and I learned to sign there,
even teaching it to an autistic girl everyone said was nonverbal. Until she
signed.
Before that all I knew was high school Spanish.

Someone told me the more languages you know, the easier it is to learn more
of them. I think that's true. Just this morning I was able to speak Spanish
with my new neighbor. He was very tolerant of my stumbling around.

Perhaps you know this already, but if not it's an eye opener. Until digital
video and even now, many European films are transferred to our standard too
fast. This comes from analog systems running 25 frames per second video vs
30fps here. Now everyone can do 24fps, but many French films we watch here
are still 25fps transfers.

That means they're 4.1% too fast. Which is just fast enough to make
understanding French more difficult. Not the same as how people talk IRL.

I've transferred and mastered many European films for US home video release
and this used to drive me crazy.
I tried to find Astrid to stream. Where did you get it?
Many thanks
Bret

[R Neil Haugen]

We get it streamed through Amazon Prime, paying extra for a PBS coverage. it's a current French TV show. Astrid is very visibly autistic (not all of us are!) and an archivist in the Paris police. And becomes part of a police homicide investigation unit.

[Bretacious2]

Ah, I get it.
Just found it.
Did you know it's Dutch?
It says 4 seasons.
I will watch it, thanks for mentioning it.

I have a good internet friend on the spectrum. Like me, he was a sign
language interpreter.
Actually we're texting right now.

If you have any links to other caption/subtitle discussions, please let me
know.

Have a great weekend

[R Neil Haugen]

Produced in Paris, the language used is clearly French. The actors and show credits are all French. On our TV.