dictation-toolbox/dragonfly

Discussion related to the Dragonfly speech recognition framework

Dane Finlay

@Danesprite

Cancelling or invalidating recognitions is a little different than asynchronous actions that can be interrupted. I haven't looked into this, but I guess you would do it with threads and signals.

Hawkeye Parker

@haughki

OT: on Windows, looking for a lightweight (not MSWord) Select and Say editor, multiple undo (not Notepad), doesn't always revert to double-spacing (not WordPad). Just want to make sure I'm not missing something obvious.

This is for Dragon.

Alex Boche

@alexboche

I think there was a thing called Dagonpad that came with Dragon. But for prose writing, mostly Dragon native select and say is just worse than a combination of sophisticated navigation commands and eyetracking. Eyetracking is probably best for long jumps then can use commands to move within a line or small number of lines. Imitation select and say (wolfmanstout accessibility api approach/ocr, Caster, or Talon probably has something). Others can elaborate.

Dane Finlay

@Danesprite

@haughki As Alex mentioned, DragonPad might be what you want. You can open it via the Dragon Tools menu or by saying "open DragonPad".

Hawkeye Parker

@haughki

@alexboche @Danesprite Tx: For some reason, I gave up on DragonPad a long time ago. I'll try again and maybe it will be great, or maybe I will remember why I stopped using it before :)

@alexboche I hear you. I don't use eye tracking or ocr, but for coding, I have a ton of navigation commands -- AceJump, gotoline, scanline, etc. But, for prose dictation, some apps are terrible: slack, gmail -- those are the two I can think of off the top of my head -- often insert double letters during dictation (at least for me), and other weirdness. Other than that, I don't have any solid data, but it feels to me like Select-and-Say applications have overall better recognition, even if I don't use any of the "correct" commands. I do find the "insert before/after" commands to be really useful to quickly get the cursor exactly where I want. Hrm. Maybe it's time to try eye tracking and/or ocr.

Ryan Hileman

@lunixbochs

often insert double letters during dictation

that's a dragon bug

they send multiple keydowns but only one key up, I think it's a timer that fires too fast or something

Dane Finlay

@Danesprite

Ah, fair enough. I don't use DragonPad either. I get by just fine with Emacs. It doesn't have Select-and-Say support though.

James Stout contributed some accessibility / Select-and-Say functionality to Dragonfly a little while ago. You can read about it here: https://handsfreecoding.org/2018/12/27/enhanced-text-manipulation-using-accessibility-apis/

He also has a few posts on and code for OCR + eye tracking if that is something you want to look into: https://handsfreecoding.org/2020/07/25/say-what-you-see-efficient-ui-interaction-with-ocr-and-gaze-tracking/

Alex Boche

@alexboche

Yeah I suppose Dragon select and say has some advantages. The fact that it uses the words on the page to recognize is nice. The annoying problem for me is that you have to pause before and after. I poured my soul in a clipboard select and say approach in Caster that works in almost any app; I don't know if anyone ever used it :(
I actually felt making corrections was occasionally worth doing for common phrases (if you can manage to not have your profile randomly deleted!).
I think there was a rumor that notepad++ was select and say but it didn't seem to work for me last time i checked.
When I was on Windows I found the speech productivity dictation box pretty cool. Rudiger had another type of dictation box with seemless auto transfer that seemed cool.
Its always worth seeing what people are up to on talon slack. People have ideas there.

Anatole Matveief

@amatveie

@haughki over on the know-brainer forum, in addition to the Dragon capture which Rudiger built and is mentioned by Alex, there's another individual who has developed an alternative to the dictation box that gets pretty good reviews https://www.knowbrainer.com/forums/forum/messageview.cfm?catid=25&threadid=35489&enterthread=y . It might be worth checking out. Select-and-Say definitely does a better job during dictation. In particular it gets the spacing right and usually gets the capitalization rate as well which is a big help. With regards to Notepad++ there is a hack for the registry hack, also documented in 1 of those forum threads, that allows it to get some Select-and-Say functionality, but I don't think it's perfect. If you are adventurous and able to install natlink you could also try Mark's vortex code which will generally enable Select-and-Say for more apps. https://github.com/mdbridge/Vocola-2/tree/vortex . I'm surprised that you have a hard time with Gmail as it seems to work pretty well with me in both chrome and edge. For Slack what I do is use it inside of the web browser, which allows for Select-and-Say vs the Windows app which does not. (I actually go one step further and have it as an 'App' in my taskbar from MS Edge. (Just click on the ... menu->Apps->install the site as an app). In other news there's an effort underway trying to get better dictation support for other apps and speech engines by using the accessibility APIs. But that's certainly not ready for prime time.

Hawkeye Parker

@haughki

@amatveie @alexboche Thank you. I have tried Rudiger's app (Dragon Capture) and SP Pro. Neither of them did exactly what I wanted. If I remember correctly, I hit some crashing bugs with SP. I can't remember exactly what it was with Dragon Capture, but it was a show stopper for me, at least.

Gmail/slack: I think you're talking about the Dragon browser plugin? As I remember, that was truly horrible: resource hogging, crashing. But, I'm on Chrome. I'm so deep in the Google ecosystem, that I'm not sure I could switch to MS Edge -- but, if the plugin really works that much better there, it might be worth it.

James Stout's accessibility API work looks amazing, but more than I want to tackle right now; similar feelings about Vortex.

Honestly, and Notepad works well for me %99 of the time: it's superfast, super lightweight, super stable. It's quick for me to switch back and forth, and I have a "transfer <app>" command which just copies everything and focuses the target application.

So, DragonPad may be it for me for now. Hilarious.

Hawkeye Parker

@haughki

So, just to wrap this thread up (on my end) I've been using DragonPad for all of 30 minutes, and it's buggy as hell in my environment. It's inserting random characters at the end of sentences. If I have more than a few lines of text with some newlines, and I go back and edit "above" in the text, sometimes it just starts inserting my dictation at seemingly random places later in the document. I'm not kidding. So, nonstarter for me.

Anatole Matveief

@amatveie

@haughki Yes DragonPad was completely non workable the last time I tried it. You can also look at James's solution for using notepad as the dictation box https://handsfreecoding.org/2015/08/30/avoid-the-dictation-box/ also note that if the standard Dragon dictation box takes a long time to load up, you could experiment with the keys to send for copy and paste in the settings. I have found that using the Windows copy and paste commands directly versus simulating ctrl+c tends to bring it up quite a bit faster, but will fail in certain applications. With regards to edge versus chrome, in my experience chrome was becoming such a memory hog/CPU hog that was unworkable. For whatever reason the browser plug-in hasn't really given me many problems although it did years ago, so maybe they've been ironing out the bugs? I'm on Dragon 15.61 and I vaguely remember the release notes talking about the browser especially edge since they made the recent change to chromium as the underlying rendering engine. Also maybe there's a setting to do this in chrome, but I find that edge is pretty adamant about putting background tabs to sleep which really helps.

tripfish

@tripfish

@haughki If nothing helps, you can also create a simple dictation box windows application. The RichTextBox element also supports Select-and-Say. I tried it a long time ago. Couldn't find any problems. But I haven't tested it for long.

James Stout

@wolfmanstout

PSA: UI Automation is backwards-compatible with IAccessible2 as of the May 2019 Windows 10 update:
https://docs.microsoft.com/en-us/windows/win32/accessibility/accessibility-whatsnew

This means that my Dragonfly integration with IAccessible2 could be completely replaced with pywinauto and would have greater compatibility. Without having dug into pywinauto, I'm guessing this would be a change for the better from a code cleanliness standpoint because the library I'm currently relying on (pyia2) is very janky.

8 replies

Quintijn Hoogenboom

@quintijn

What is wrong with DragonPad? (For Dragon users)... There is also still WordPad, a light weight word version around. I shall try to revive my Unimacro grammar "_edit.py", https://qh.antenna.nl/unimacro/grammars/globalgrammars/edit/index.html. So: not working at the moment. The idea was always to switch to DragonPad (or other program with full dictation control), either empty, or with a paragraph copied, or with the whole original document copied. With "edit ready" the text is pasted back in the original window. I don't know why I left this grammar, it worked quite nice.

6 replies

LexiconCode

@LexiconCode

What kind of dictation would cause the following error?

INFO:dictation.word_parser_factory: Selected word parser class WordParserDns11 because natlink.getWordInfo('.\\dot') returned None

Reviewing the dragonfly code it looks like it's an issue handling Dragon special formatting?

1 reply

Dane Finlay

@Danesprite

Dragonfly2 version 0.31.2 has been released, as of a few days ago. A number of issues with keyboard input on Windows are fixed in this version.

If anyone has had trouble with the Key or Text actions not working properly on Windows, then the new version will probably fix it. This will be relevant mostly to those using keyboard layouts other than QWERTY.

LexiconCode

@LexiconCode

Thanks Dane! That's great!

MeteorVsBunny

@MeteorVsBunny

Key("apps") does not work for me in 31.2, it does work in 31.1
I'm using Windows, caster and Dragon

Dane Finlay

@Danesprite

@MeteorVsBunny Thanks for reporting this! Clearly I should have tested my changes a bit more! Apparently that key, along with some others, is what is called an extended key. I will fix this in the next version.

Dane Finlay

@Danesprite

@MeteorVsBunny That is fixed on the master branch now if you need it, though I'll release version 0.31.3 in the next day or so.

Hopefully this is the last thing wrong with Dragonfly's Windows keyboard code. 🤞

MeteorVsBunny

@MeteorVsBunny

thanks, that was fast

Dane Finlay

@Danesprite

No worries. Well, it wasn't a difficult fix. :-)

tieTYT

@tieTYT

if I use the dictation box and correct the text, then cancel, does that still help the ai?

LexiconCode

@LexiconCode

if I use the dictation box and correct the text, then cancel, does that still help the ai?

My guess the data for the correction is saved before you say cancel to the closeout box.

Is there a way to pass a parameter through function context?

Dane Finlay

@Danesprite

Is there a way to pass a parameter through function context?

Well, you can pass default values FuncContext(function, text=""). What is your use case?

Dane Finlay

@Danesprite

@LexiconCode Ah, okay then. You could have a get_grid_mode_context() function for this:

def get_grid_mode_context(grid):
    return FuncContext(lambda: get_grid_mode(grid))

LexiconCode

@LexiconCode

Thanks!

Dane Finlay

@Danesprite

:+1:

LexiconCode

@LexiconCode

Working with function context I had a thought. This could be true all context Classes.
Perhaps it could be a mode parameter to the base Context classes. Based on the type of mode dictates when that context is processed.
timer (Seconds)
call Something similar to get_engine().process_grammars_context() But just for those that are call not all Context and it would only be processed when the function is called.
utterance - The default mode - process every utterance

With timer and call It might allow for functions that you normally would not want to process per utterance due to performance reasons.

LexiconCode

@LexiconCode

Although I acknowledge though these features could be created with function context to achieve similar functionality.

Dane Finlay

@Danesprite

You could implement this with a few Context sub-classes to be used instead. It seems unnecessarily complex to me.

tieTYT

@tieTYT

I have never ever ever been able to train DNS to correct a word or phrase if, when I say it, the list of corrections doesn't include that word as a choice. Is this just the way DNS works? I don't understand the point of "spell that" since it's never done anything useful that I can notice

Dane Finlay

@Danesprite

@tieTYT It sounds like this is just the way DNS works and you have to train misrecognised words separately using "train word".

Hawkeye Parker

@haughki

@tieTYT You might try asking over at https://www.knowbrainer.com/forums/forum/index.cfm. Lot of hardcore dragon users over there. When I add a new word to the vocabulary, I always train it, but I have no clear sense that it really helps. I gave up on "spell that" a long time ago.

tieTYT

@tieTYT

@tieTYT It sounds like this is just the way DNS works and you have to train misrecognised words separately using "train word".

Does that usually fix the problem? Let me give an example, I've never been able to get Dragon to type Eva's. Even after training.

I can get it to say Eva, but Eva's comes out as even as

LexiconCode

@LexiconCode

@tieTYT It sounds like this is just the way DNS works and you have to train misrecognised words separately using "train word".

Does that usually fix the problem? Let me give an example, I've never been able to get Dragon to type Eva's. Even after training.

Did you try dictating it and training it in a sentence rather than an individual word?

tieTYT

@tieTYT

The former, yes. When you say training, you mean training a whole phrase in train word?

If yes, do you add this whole phrase to your vocabulary editor?

LexiconCode

@LexiconCode

Oh I meant to say the correct that Instead of the train word No as a individual word but the surrounding sentence gives Dragon correct that the context of how and when It should be recognized. I've only found the train word to be effective for small changes in pronunciation. For words that sound similar I've had to train them as well.

tieTYT

@tieTYT

ah. Well, as i said above,

I have never ever ever been able to train DNS to correct a word or phrase if, when I say it, the list of corrections doesn't include that word as a choice.

And that is often the case with sentences

tieTYT

@tieTYT

I trained a sentence without adding it to the vocabulary editor, and that seems to have fixed it when I say the sentence now?

LexiconCode

@LexiconCode

I trained a sentence without adding it to the vocabulary editor, and that seems to have fixed it when I say the sentence now?

I don't think I've ever tried that. Not sure

Dane Finlay

@Danesprite

I think some of you in this channel might like this example command module. It demonstrates window movement with Dragonfly's Window.move() method, including its optional argument for animating window movement.

https://github.com/dictation-toolbox/dragonfly/blob/master/dragonfly/examples/_window_movement_example.py

The available commands are as follows:

move [this] window here [animated]
center [this] window here [animated]
move notepad [window] here [animated]
center notepad [window] here [animated]

Where communities thrive

People

Repo info

Activity