CODEWITHSUNDEEP
c#unity-game-enginespeech-recognition
user3871
user3871

Reputation: 12708

Speech to text and text to speech for foreign languages

I'm considering porting a speech 2D HTML5 web game I've built to Unity2D for iPhone and Android. I'm a full-stack web developer, and not a Unity developer, so an agency would help me build the Unity app. Before signing with them, I need to be sure both Speech to Text (STT) and Text to Speech (TTS) services are available for Mandarin, Spanish, and English, otherwise I'd waste a lot of money up front.

For Web, Webkit Speech (STT Docs, STT Demo, TTS Docs, TTS Demo) is easily accessible via the browser. I've found that IBM Watson has an API available, and has demos for STT and TTS, and I've found that they have a Unity SDK here, but I don't have the skillsets to test the Unity SDK.

I'm looking for guidance on great STT and TTS APIs that the agency can use for those three foreign languages.

  1. Does the Unity SDK provide support for frontend STT and TTS audio streaming? STT needs to capture users' voice input and transcribe it quickly. Likewise, TTS needs to allow the user to hover over a target language word and listen to a near-native pronunciation.
  2. Does it offer both STT and TTS for Spanish, Mandarin, and English?
  3. What other NLP APIs are there which meet my requirements?

Apologies, I'm completely new to Unity/phone development so any guidance here would be extremely helpful. If no APIs exist that meet these requirements then Unity won't work for my app since STT and TTS is critical.

Upvotes: 2

Views: 1254

Answers (1)

Nikolay Shmyrev
Nikolay Shmyrev

Reputation: 25210

Overall, realtime audio recording in Unity is awful, the system is simply not designed to record audio continuously. You can record a clip with AudioSource but that is a clip of fixed length, not a streaming solution.

For streaming you can get the audio with AudioFilterRead but it is not really the API for recording, it is more for effects. For recording it has unpredictable latency and also slows down the UI significantly.

As a result, you can only have push-to-talk kind of interaction, not realtime interaction.

If you have other alternatives you'd better consider them too. For example, you can consider native app.

Upvotes: 2

Related Questions