Nvidia's Neotron 3.5: Faster, Smarter Speech-to-Text

Summary

Nvidia has released Neotron three point five, a streaming automatic speech recognition model that can potentially replace your entire speech-to-text setup. This 600 million parameter model supports forty languages from a single checkpoint and offers significant speed improvements by reusing previous computations, making it up to seventeen times faster on high-end hardware. Unlike older models that reprocessed overlapping audio chunks, Neotron three point five uses 'key-value caching' to efficiently integrate new audio. Users can adjust inference settings like chunk size to balance latency and accuracy, from eighty millisecond chunks for near real-time transcription to over one second chunks for phrase-level output. While punctuation and capitalization can be inconsistent in streaming mode, the model excels at supporting multiple languages, with nineteen languages working out of the box, thirteen at a production level, and eight more adaptable through fine-tuning. A key feature is word boosting, a decode-time technique that increases the likelihood of transcribing specific rare words or phrases like product names or surnames without retraining the model. This is achieved by adding custom words or phrases to a boosting tree, guiding the model's predictions. The model also supports speaker diarization for identifying different speakers in an audio stream, either through external models or Nvidia's Nemo framework, which can be particularly useful for podcasts. The speaker's embeddings can also be captured for more accurate speaker attribution. Overall, Neotron three point five is presented as a powerful and accessible tool for various speech-to-text applications, enabling local hosting and customization.