Source Media and Learning News April 2019 issue

People with a hearing impairment can benefit from captioning of speech and sounds in videos and people with a visual impairment can benefit from audio description of images and actions in videos. Captioning and audio description can also be useful for anyone unable to hear or see a video for environmental reasons (e.g. sound to quiet or too much background noise or vision is obscured). This article discusses how speech recognition captioning with collaborative editing can provide affordable transcription/captioning of lecture recordings and educational videos and discusses the potential of Artificial Intelligence for audio description and sign language translation.

Many countries have had long standing regulations about captioning or audio description of broadcasted television programs but these don’t normally apply to online video. The UK’s Equality Act 2010 requires universities to make anticipatory reasonable adjustments and so universities should caption all their lecture recordings rather than only caption a lecture recording if requested by a deaf student.Paying captioning companies commercial rates for captioning lecture recordings is not affordable by universities but automatic speech recognition captioning is affordable.

One of the arguments that some of the captioning companies use to persuade customers to use their manual captioning service is to say that speech recognition automatic captioning although much cheaper, produces inaccurate captions. Although background noise and music and poorly positioned microphones can reduce the accuracy of automatic speech recognition, it can be as accurate as human transcribers, especially if students are used to collaboratively correct caption errors.

Unfortunately no lecture capture company’s technology currently enables collaborative editing.

Synote was developed to enhance learning from lecture recordings through collaborative editing of speech recognition transcription with additional features including synchronised notes, clip playlists and a print friendly audio only version as shown in the following screen captures.

It is important for teachers to make a good quality recording and this can be achieved by wearing a wireless microphone and adjusting the recording level to provide a good signal to noise ratio. If a teacher uses a fixed lectern microphone and turns or moves away from the microphone to write on the board or walk around the room then the recorded speech level and signal to noise ratio will decrease. If the lecturer repeats any questions or comments or answers from the students then the speech of the students does not need to be transcribed. It is also possible to record and transcribe the speech of the students using a wireless microphone, either handheld or passed around or using an app on a mobile phone.

As the quality of the recording worsens then speech recognition may struggle more than human transcribers and a solution to this problem of improving the accuracy of speech recognition transcription is to use students to collaboratively correct errors. Students correcting errors engage more strongly with the lecture content to improve their learning and so justifying the awarding of academic credit for correcting errors. Universities and students could choose appropriate rewards.

Students correcting errors in their own lectures involves little extra effort while they listen, watch, and read the recording and captions as it is difficult not to notice errors. They also generally know the subject better than a professional captioner as captioning companies do not guarantee to provide a specialist in that subject. Images and slides can be automatically synchronised with the transcript to enable printing out of all of the information from a lecture.

Artificial intelligence machine learning using Deep Neural Networks has proved very powerful for improving the accuracy of speech recognition and recognising images. AI can also be used for enhancing speech recognition through automatic lipreading and can also recognise some sounds on videos for captioning (e.g. Applause, music and laughter). AI can also modify lip patterns and faces on videos to make them easier to lipread.

AI cannot however currently provide automatic audio description of videos as this requires reasoning and understanding subtle meanings and context to identify what visual information is important. For example in a video if a person leaves the room then while it might be possible for the AI system to identify this had happened this might only be important information to provide in an audio description if it was necessary for the viewer to know that the person leaving the room could not have heard what was being discussed in the room after they had left the room.

AI can also provide automatic sign language translation of the captions using avatars although the quality of translation for a visual language is not as good as translations between written languages which have vast amounts of data available for training the AI systems.

Mike is one of the keynote speakers at the Media & Learning Conference on Video in High Education and will be speaking on the topic of accessibility on 6 June in Leuven.

Author

Prof. Mike Wald, University of Southampton, UK

Mike’s background

More about Mike’s research activities. Mike will give a keynote presentation at the Media & Learning Video in Higher Education Conference on 5-6 June 2019

---