Innovation - The technologies we use to drive meaningful progress

Asset 72@1.png

Designing a data collection app for citizen science

In order to obtain the 600 samples we would need to test and train the app, we needed to design and develop an app that could be easily deployed on the AppStore and PlayStore and downloaded by parents around the country. 

The app needed to be able to work offline but to upload the data when online later and work on a range of mobile phones and tablets so that we could have participants from all backgrounds involved. 

Instructions needed to be clear for parents and for children and we needed to be able to quickly explain the purpose of the project and how their participation would make a difference!

Designing a data collection app for use by children

We worked with children, parents and clinicians to design an app that could be used to elicit a story retelling sample from children remotely.  We needed to make sure that the theme was universally recognised and that boys and girls would be engaged.

After lots of work with children, a treasure quest was decided to be fun and internationally appropriate.  The main characters of a boy and a dog are familiar and the supporting cast of a tortoise and birds were also identifiable for children. 

We tested the voices and the sound effects, the rewards and the colours - to make sure the app would be motivating for children starting at four and going right up to 8! 

Asset 42@1.png
Asset 52@1.png

Speech Recognition Technology

Speech recognition technology is used in all aspects of our day to day life now. It is one of the core technologies in use in this project as the child records their story retell into the app and it needs to be transcribed so that the analysis of language can take place. 

Transcription improvement tool

After the automated transcription is presented to the clinician, they can edit any errors the speech recogniser has made and make sure it is sorted into separate utterances. 

The user interface is designed to make this as quick a task as possible and allows you to listen back to the recording as you go through.  Marking up any mispronunciations and mazes also improves the accuracy of the grammatical analysis that the tool will undertake. 

Asset 41@1.png
Asset 76@1.png

Language analysis

Using the confirmed transcription, the tool analyses linguistic features in the sample. It looks at the language features that are expected to have been acquired at different ages. It also looks at the macrostructure elements of the story. 

The accuracy of the calculation of the features by the software has been compared to that of the Newcastle University researchers and features needed to meet an acceptable agreement score to move forward to the clinical version of the tool.

Acoustic analysis

In addition to looking at the language features of the samples, we also look at the acoustic properties of the recording. That includes looking at the nature and duration of pauses as well as the balance of talking between the child and the adult.

We can also look at the pitch, loudness and other variables of the child's spoken sample.

Asset 53@1.png
Asset 54@1.png


With over 100 features being extracted and calculated by the app, it is important that the information is presented in a logical way for clinicians to interpret so that they can make decisions about each child. 

The report provides data on the performance of each child and shows how it compares to the performance of children of the same age, based on the data we collected in our citizen science project in 2020. 

This helps the clinician to understand more about the profile of language skills for their client to support their decision making. 

Remote integration

Our project started in November 2019 and by the time we were midway through our citizen science work package we knew we needed to make sure that this language analysis tool could be used in remote speech and language therapy sessions using technology to not just mirror a screen, but to really take the therapist into the child's app. 

Our clinical evaluation will now include testing the app in both face-to-face and remote sessions, which will be evermore important as we go forward.

Asset 60@1.png
Asset 92@1.png

Research tools

A group of UCL researchers in the Neurotherapeutics group piloted the transcription improvement tool and analysis software to support the transcription and analysis of 66 x 2-minute picture description samples. The team calculated that using our software tools would save them 20% of their time when compared to conducting the work manually. The researchers also identified the benefits of the tool beyond time-saving:

* reducing the differences between scorers

* having access to the output of the analysis in a file ready to copy into a group document

* having access to measuring features that take considerable time to do by hand - such as counting the occurrence of verbs or nouns.