Using the Speech Assistant without clicking with the mouse

The Herbi Speech Assistant has been updated to explore the idea of having it speak, without you having to click words or phrases with the mouse. It is likely that more updates are required in order for this feature to be usable in practice, but these updates can be made as feedback is provided. To try out this feature, follow the steps below. Note that it is necessary to click with the mouse to set up the feature where clicking is not later required to speak.

1. Install the latest version of the Speech Assistant. The latest version can be found here. If you have an earlier version of the Speech Assistant software already installed on your computer, you may need to uninstall that version first. Once you've installed the latest version, you can run it by double clicking on the shortcut on the desktop.

The Speech Assistant may look similar to this:

The Speech Assistant software


2. It's likely that you're interested in typing specific letters, so click the "Show keyboard" check box near the top of the window. The keyboard will then appear.

Speech Assistant showing the keyboard


3. As you type letters from the keyboard later, you'll want the contents of the lists to be filtered as you type. So click on the "Settings" button. (Future improvement: A way to scroll through the lists without having to click with the mouse will be added.) The Settings window will then appear. Click on the "Lists" tab near the top of the Settings window.


Then click on the box that says "Filter the items shown in the lists as you write". Once you've done that, click the "Screen" tab near the top left of the Settings window.


4. You're then back at the main Settings for the Speech Assistant. You'll notice a "Hover" setting near the bottom right of the Settings window. This is used to say how long the mouse cursor needs to be hovering over something shown in the Speech Assistant, before the Speech Assistant will automatically click it for you. By default, the setting for this is "No hover", which means that the Speech Assistant will not automatically click anything. 


So click on the button that says "No hover". A list will appear showing times.

Speech Assistant showing dwell time list

Pick the time delay you'd like the Speech Assistant to use when automatically clicking buttons for you. For example, if you clicked "2 seconds", the Settings window would look like this:

Speech Assistant showing a dwell time of 2 seconds

You can change the time later if you want to. Then click the OK button.


You've now set up the Speech Assistant to speak without you having to click with the mouse. If you hover the mouse over any of the following buttons for the time you selected in the Settings window, the button will automatically be clicked:

Blue buttons which can be automatically clicked.

You can also have any of the words or phrases shown in the lists automatically clicked. Once an entry in a list has been automatically clicked, you need to move the mouse out of the list and back into the list in order to automatically click anything from the list again. (This avoids list entries from being automatically clicked repeatedly.) So you can hover over the following, to automatically click it.

Lists of words and phrases

You can also hover over the blue keyboard keys to click them.

Speech Assistant's blue keyboard keys


Given that the lists can't be scrolled through today by hovering with the mouse, it is necessary to have the lists filtered to show the phrase you want. So you can hover of the blue keys to start typing. Say you hover over the letter 'g' to automatically click it.

Speech Assistant after the letter g is clicked.

All the lists will then be filtered to only show words and phrases that begin with 'g'. So if you then hover over the 'o' key to have it automatically clicked, all the lists would only show words and phrases that being with 'go'. Once a list shows the word or phrase you want, you can hover the mouse over that entry in the list, and have it automatically clicked. If the phrase is in the Favourites lists, then it will be spoken immediately. Words and phrases in the other lists will be added to the white area below the list so you can build up a bigger phrase.

You may see a word or phrase shown in the yellow suggestion area as you have the letter keys clicked. You can hover the mouse over any of those suggestions, they will also be automatically clicked.

(If the word or phrase you want never shows in the lists or in the yellow suggestion area, the lists of known words and phrases can be updated to include it. But today this does require the use of a mouse to click.)


Whenever you hover over mouse cursor over a button to have it clicked automatically, a little progress bar will appear near where the cursor was when it moved over the button. It might look something like this:

An empty progress bar which appears over a button which can be clicked automatically.

As you hover the mouse cursor over the button, the progress bar fills up.

Hover progress bar filling up.

When the hover time has elapsed, the progress bar goes away and the button is automatically clicked. (On the fastest hover time, the progress bar might not actually appear to get filled up before the button is clicked.)



Other future improvements

While the Speech Assistant can be used today without clicking with the mouse as described above, there are a number of changes that could be made in order to make this feature more usable.

1. All the buttons and suggestions on the screen are small, so it's difficult to see what they are. It would be useful for these things to be bigger, particularly on a large computer screen. On small computer screen, the balance is making things bigger while still showing everything that needs to be shown.

2. It may be difficult to keep the mouse still and so it might move away from the thing you want to hover over. This is made more difficult given that many of the things shown on the screen are so small. It may be possible to update the software such that during the time required to automatically click the button or phrase, the mouse can move briefly way from the button or phrase to be clicked.

3. The predictions shown aren't really sufficient to use the Speech Assistant without handwriting. So the prediction list will have to be expanded considerably before the Speech Assistant is useful without handwriting.


While many common words need to be added to the predictions lists, the phrase lists have been recently updated with some common phrases. Some of these phrases are complete short sentences, and others would have additional words added to the end of them before they're spoken.

The results shown in the picture below, (using a large computer screen), appeared after the following actions:

- Set up the Speech Assistant app to have buttons clicked automatically, and to show the on-screen keyboard, and to filter the lists as letters are clicked.

- Hover over the letter 'i' to click that key. When it gets clicked automatically, the yellow suggestion area shows some of the phrases beginning with 'i' that have been spoken recently on that computer. The Phrase list shows all the phrases that the app knows which begin with the letter 'i'.

- Hover over the phrase starting "I have a speech problem" in the Phrase list until it gets automatically clicked. In that case, the phrase gets copied down to the area above the yellow area.

- The phrase can either be spoken then by hovering over the "Speak" button, or added to the Favourites list by hovering over the yellow '*' button. 

Lists filters starting with the letter 'i'