79

The short version of the question: I am looking for a speech recognition software that runs on Linux and has decent accuracy and usability. Any license and price is fine. It should not be restricted to voice commands, as I want to be able to dictate text.


More details:

I have unsatisfyingly tried the following:

All the above-mentioned native Linux solutions have both poor accuracy and usability (or some don't allow free-text dictation but only voice commands). By poor accuracy, I mean an accuracy significantly below the one the speech recognition software I mentioned below for other platforms have. As for Wine + Dragon NaturallySpeaking, in my experience it keeps crashing, and I don't seem to be the only one to have such issues unfortunately.

On Microsoft Windows I use Dragon NaturallySpeaking, on Apple Mac OS X I use Apple Dictation and DragonDictate, on Android I use Google speech recognition, and on iOS I use the built-in Apple speech recognition.

Baidu Research released yesterday the code for its speech recognition library using Connectionist Temporal Classification implemented with Torch. Benchmarks from Gigaom are encouraging as shown in the table below, but I am not aware of any good wrapper around to make it usable without quite some coding (and a large training data set):

System Clean (94) Noisy (82) Combined (176)
Apple Dictation 14.24 43.76 26.73
Bing Speech 11.73 36.12 22.05
Google API 6.64 30.47 16.72
wit.ai 7.94 35.06 19.41
Deep Speech 6.56 19.06 11.85

Table 4: Results (%WER) for 3 systems evaluated on the original audio. All systems are scored only on the utterances with predictions given by all systems. The number in the parentheses next to each dataset, e.g. Clean (94), is the number of utterances scored.

There exist some very alpha open-source projects:

I am also aware of this attempt at tracking states of the arts and recent results (bibliography) on speech recognition. as well as this benchmark of existing speech recognition APIs.


I am aware of Aenea, which allows speech recognition via Dragonfly on one computer to send events to another, but it has some latency cost:

enter image description here

I am also aware of these two talks exploring Linux option for speech recognition:

6
22

Right now I'm experimenting with using KDE connect in combination with Google speech recognition on my android smartphone.

KDE connect allows you to use your android device as an input device for your Linux computer (there are also some other features). You need to install the KDE connect app from the Google play store on your smartphone/tablet and install both kdeconnect and indicator-kdeconnect on your Linux computer. For Ubuntu systems the install goes as follows:

sudo add-apt-repository ppa:vikoadi/ppa
sudo apt update
sudo apt install kdeconnect indicator-kdeconnect

The downside of this installation is that it installs a bunch of KDE packages that you don't need if you don't use the KDE desktop environment.

Once you pair your android device with your computer (they have to be on the same network) you can use the android keyboard and then click/press on the mic to use Google speech recognition. As you talk, text will start to appear where ever your cursor is active on your Linux computer.

As for the results, they are a bit mixed for me as I'm currently writing some technical astrophysics document and Google speech recognition is struggling with the jargon that you don't typically read. Also forget about it figuring out punctuation or proper capitalization.

enter image description here

enter image description here

3
  • 8
    The problem with google is it's not text to speech, it sends it back to google. This is bad for privacy. – Owl Dec 12 '19 at 15:33
  • After struggling with audio-to-text utilities on Linux for a long time, I solved the problem with a trivial hack: just play the audio over my laptop speakers and put my phone next to it, with Google Docs in text-to-speech mode. Stupid but it worked :) – Radon Rosborough Mar 7 '20 at 0:34
  • 1
    I am surprised that this is still the "best" answer, and continues to slowly accumulate votes. – shockburner Jan 13 at 17:58
4

As one more Linuxer searching for a useful speech-to-text (dictation) program, I took a look into speechpad.pw:

  • it recognizes my mother tongue very well
  • it works fast and very reliable

Downsides:

  • of course it is proprietary and closed software from Google
  • a Google service will listen to, process and supposedly store every word you speak
  • audio and text will be processed and obviously stored by Google
  • speechpad.pw requires a monthly / quaterly / yearly subscription fee
  • speechpad.pw only runs as an addon to Google Chrome browser - no other browser

So, speechpad.pw is very proprietary and also closed source and also bound to Google which we all know as a sleepless meta data, personal information and personal contents collector.

These downsides make it a no-go application for me though the speech recognition itself works very well - much better than anything else I have seen so far.

2
  • Thanks, yes significant downsides, especially that it only works in the Chrome browser. – Franck Dernoncourt Oct 28 '16 at 22:45
  • 2
    You could use Google Docs on Chrome and use their "Tools" » "Voices Typing ..." option. Probably exact same speech recognition software, but it's free. Then copy paste the results from your doc to wherever you need the text. – Alexis Wilke Nov 10 '17 at 20:19
4

vosk-api

https://github.com/alphacep/vosk-api/

It supports 7+ languages.

First you convert the file to the required format and then you recognize it:

ffmpeg -i file.mp3 -ar 16000 -ac 1 file.wav

Then install vosk-api with pip:

pip3 install vosk

Then use these steps:

git clone https://github.com/alphacep/vosk-api
cd vosk-api/python/example
wget https://alphacephei.com/kaldi/models/vosk-model-small-en-us-0.3.zip
unzip vosk-model-small-en-us-0.3.zip
mv vosk-model-small-en-us-0.3 model
python3 ./test_simple.py test.wav  > result.json

The result is stored in JSON format.

The same directory also contains an SRT subtitle output example, which is more human readable and can be directly useful to people with that use case:

python3 -m pip install srt
python3 ./test_srt.py test.wav

The sections below show some testing I did with it.

test.wav case study

The test.wav example given in the repository says in perfect American English accent and perfect sound quality three sentences which I transcribe as:

one zero zero zero one
nine oh two one oh
zero one eight zero three

The "nine oh two one oh" is said very fast, but still clear. The "z" of the before last "zero" sounds a bit like an "s".

The SRT generated above reads:

1
00:00:00,870 --> 00:00:02,610
what zero zero zero one

2
00:00:03,930 --> 00:00:04,950
no no to uno

3
00:00:06,240 --> 00:00:08,010
cyril one eight zero three

so we can see that several mistakes were made, presumably in part because we have the understanding that all words are numbers to help us.

Next I also tried with the vosk-model-en-us-aspire-0.2 which was a 1.4GB download compared to 36MB of vosk-model-small-en-us-0.3 and is listed at https://alphacephei.com/vosk/models:

mv model model.vosk-model-small-en-us-0.3
wget https://alphacephei.com/vosk/models/vosk-model-en-us-aspire-0.2.zip
unzip vosk-model-en-us-aspire-0.2.zip
mv vosk-model-en-us-aspire-0.2 model

and the result was:

1
00:00:00,840 --> 00:00:02,610
one zero zero zero one

2
00:00:04,026 --> 00:00:04,980
i know what you window

3
00:00:06,270 --> 00:00:07,980
serial one eight zero three

which got one more word correct.

IBM "Think" Speech case study

Now let's have some fun, shall we. From https://en.wikipedia.org/wiki/Think_(IBM) (public domain in USA):

wget https://upload.wikimedia.org/wikipedia/commons/4/49/Think_Thomas_J_Watson_Sr.ogg
ffmpeg -i Think_Thomas_J_Watson_Sr.ogg -ar 16000 -ac 1 think.wav
time python3 ./test_srt.py think.wav > think.srt

The sound quality is not great, with a lot of microphone hissing noise due to the technology of the time. The speech is however very clear and paused. The recording is 28 seconds long, and the wav file is 900KB large.

Conversion took 32 seconds. Sample output of the three first sentences:

1
00:00:00,299 --> 00:00:01,650
and we must study

2
00:00:02,761 --> 00:00:05,549
reading listening name scott

3
00:00:06,300 --> 00:00:08,820
observing and thank you

and the Wikipedia transcription for the same segment reads:

1
00:00:00,518 --> 00:00:02,513
And we must study

2
00:00:02,613 --> 00:00:08,492
through reading, listening, discussing, observing, and thinking.

"We choose to go to the Moon" case study

https://en.wikipedia.org/wiki/We_choose_to_go_to_the_Moon (public domain)

OK, one more fun one. This audio has good sound quality, with occasional approval screams by the crowd, and a slight echo of the venue:

wget -O moon.ogv https://upload.wikimedia.org/wikipedia/commons/1/16/President_Kennedy%27s_Speech_at_Rice_University.ogv
ffmpeg -i moon.ogv -ss 09:12 -to 09:29 -q:a 0 -map a -ar 16000 -ac 1 moon.wav
time python3 ./test_srt.py moon.wav > moon.srt

Audio duration: 17s, wav file size 532K, conversion time 22s, output:

1
00:00:01,410 --> 00:00:16,800
we choose to go to the moon in this decade and do the other things not because they are easy but because they are hard because that goal will serve to organize and measure the best of our energies and skills

and the corresponding Wikipedia captions:

89
00:09:06,310 --> 00:09:18,900
We choose to go to the moon in this decade and do the other things,

90
00:09:18,900 --> 00:09:22,550
not because they are easy, but because they are hard,

91
00:09:22,550 --> 00:09:30,000
because that goal will serve to organize and measure the best of our energies and skills,

Perfect except for a missing "the" and punctuation!

Tested on vosk-api 7af3e9a334fbb9557f2a41b97ba77b9745e120b3, Ubuntu 20.04, Lenovo ThinkPad P51.

This answer is based on https://askubuntu.com/a/423849/52975 by Nikolay Shmyrev with additions by me.

2
3

I'd recommend Mozilla DeepSpeech. It's an opensource speech to text tool. But you will need to train the tool.

You can download the pre-trained model or use Mozilla Common Voice DataSets to create your own. For very clear recordings accuracy rate is good. For my transcription projects, it was still not sufficient, as the recordings had lots of background noises, and were not of great quality.

I used Transcribear instead, a browser based speech to text tool. You will need to be connected online to upload recordings to the Transcribear server.

1
  • AFAIK Mozilla DeepSpeech only works for utterances shorter than a few seconds. – Franck Dernoncourt May 18 '20 at 17:40
3

After trying Simon and Julius on Kubuntu, which I wasnt able to install properly, I stumbled on the idea to try using Mycroft, the open source AI Assistant (competing with Google Home and Amazon Alexa).

After having the KDE Plasmoid install fail, I was able to get pretty good speech recognition going with the regular install. It has a mycroft-cli-client to view debugging messages in and a somewhat active community forum. Some of the docs are a little out of date, but I have noted that on the forum and in GitHub where applicable.

The speech rec is really pretty good and you can install Mimic, a local recognition engine. And it is cross-platform and saw an Android app I havent tried yet. My next step is reproduce some of the basic desktop shortcut commands I was hoping for in the Plasmoid, and a dictation Skill for large text fields.

https://github.com/MycroftAI/mycroft-core

https://community.mycroft.ai/

2

The Chrome App "VoiceNote II" (http://voicenote.in/) is working great on my Xubuntu 16.04 machine. No voice-training required, and set-up was simple. One search to find it, one click to install, one click to create a shortcut and to the Desktop bind it.

1
2

I'm using the KD Connect app. it is working quite effectively! I am able to keep my eyes on the monitor while speaking with the phone on the desk. The only downside is that this is being done through Google keyboard. it is neither free, native, nor open source.this comment has been posted without making any and type corrections

1

I would suggest using dragon on your phone or tablet, then emailing the text to yourself. Its a drag but it works and is very accurate. If you insist on using Linux for this, getting a second display will make life much easier to copy and past.

I haven't tried this but you might be able to use or adapt the Python Bluetooth Chat program with dragon on your tablet/phone. There may also be remote-keyboard apps for mobile devices that may support dictation input.

I shall experiment and try to get back to you with something more definitive.

Not the answer you're looking for? Browse other questions tagged or ask your own question.