Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Nov 17 14:49

    mrob95 on master

    Add core context (compare)

  • Nov 17 14:28

    mrob95 on master

    Add install script (compare)

  • Nov 01 07:33

    mrob95 on master

    Update dependencies (compare)

  • Oct 27 17:56
    mrob95 synchronize #19
  • Oct 27 17:56

    mrob95 on languages

    Improve formatting (compare)

  • Oct 27 17:03
    mrob95 synchronize #19
  • Oct 27 17:03

    mrob95 on languages

    Initial multilanguage support load to edit Further improvement and 1 more (compare)

  • Oct 27 16:50

    mrob95 on master

    Transition sublime and template (compare)

  • Oct 27 16:17
    mrob95 closed #20
  • Oct 27 16:17

    mrob95 on master

    Transition core Latex and apps Aliases and 5 more (compare)

  • Oct 27 16:16
    mrob95 synchronize #20
  • Oct 27 16:16

    mrob95 on breathe

    Generalise latex contexts (compare)

  • Oct 26 14:06
    mrob95 opened #20
  • Oct 26 14:03

    mrob95 on breathe

    Fix aliases (compare)

  • Oct 26 13:45

    mrob95 on breathe

    Getting there (compare)

  • Oct 24 19:32
    mrob95 synchronize #19
  • Oct 24 19:32

    mrob95 on languages

    Further improvement (compare)

  • Oct 24 14:56

    mrob95 on breathe

    Load aliases from file (compare)

  • Oct 23 14:32

    mrob95 on breathe

    Tidying (compare)

  • Oct 23 13:51

    mrob95 on breathe

    Aliases (compare)

Mike Roberts
@mrob95
Yeah the numerical evaluation is pretty useful for getting answers without having to type or copy stuff into another calculator program. I think it can do ODEs and stuff but that would be cheating ^^
esc123
@esc123
Yeah I use the compute function in SN frequently. It is very useful in saving the calculator grunt work
I found it particularly useful when checking my work in Linear Algebra for instance
Yeah I can meet up with the scribe a few mins before the exam but that's about it. They are hesitant for me to spend too much time with the scribe before the exam I presume so as to reduce likelihood of cheating perhaps. I was told that the scribe will have domain knowledge but other than that I'm just gonna hope for the best :D
alexboche
@alexboche

I think putting some math vocabulary in with dictation is very useful so we don't have to pause or use "say <dictation>" all the time.
Here's what I tried so far (including some of Mike's old code). Capturing dictation and then replacing some words with a dictionary (would be in a separate grammar that would be disabled unless you are doing math).

math_replacement_dictionary = {"math gamma": r"$\gamma$",
    "math gamma below beta": r"$\gamma_\beta$"}
def math_replace(dictation, dictionary):
    input_list = str(dictation).split(" ")
    output_list = []
    for i in range(len(input_list)):
        if input_list[i] == "cap":
            input_list[i+1] = input_list[i+1].title()
        else:
            output_list.append(input_list[i])
    pre_math_string = " ".join(output_list)

    output_string = copy.copy(pre_math_string)
    for k, v in sorted(dictionary.items(), key=lambda pair: len(pair[0]), reverse=True):
        output_string = output_string.replace(k, v)
    Text(output_string).execute()

"<dictation>": Function(math_replace, dictionary=math_replacement_dictionary),

The first part of the function is just to getcapitalization for regular dictation. The reason I did the sorting stuff is because I want to make sure I am replacing the largestthing first. e.g. replacing "math gamma below beta" before "math gamma" since otherwise that would cause problems.
We can generate a dictionary for the math replacement dictionary. Let me know what people think. David zurow said he had some thoughts about a better way to do this sort of thing am hoping he will share.

I still need to get it to put in spaces at the beginning of each utterance unless there already is aspace which I guess is something that Dragon does automatically but is not done by this. A variety of things seem kind of clunky when all of your dictation is passed through this. e.g. scratch that doesn't work.
@annakirkpatrick you can have backslashes in the Dragon vocabulary word if you put them in the printed form which is in preferences under a word in the Vocabulary Editor. However, if you are using the approach above if you pass everything through that command, the printed form will be lost it will just write out the written form; in other words you can't do both approaches at the same time.

alexboche
@alexboche
this command would be non- CCR
Mike Roberts
@mrob95

I'm not sure it's worth it really. Losing dragon's built-in editing features is annoying and this would be vulnerable to slight misrecognitions of the command words. I don't think pausing before starting to dictate latex wastes too much time in the scheme of things.

Probably an easier way of accomplishing something similar would be to just dictate as normal until you have a document with numerous instances of "math gamma" and then just do a find and replace for all of them.

alexboche
@alexboche

By "Dragon 's built-in editing features" do you mean the commands like "insert before <dictation>" ? That still works with this but those don't usually work in math -based apps anyways. There are a few small issues regarding spacing that need to be worked out. for now I think I'm just going to have a space at the end of the text action. but spacing with things like "open paren" works though the spacing does not seem to work properly for custom Dragon vocabulary words or for "slash". I don't think the find and replace approach would be suitable if you have a lot of different symbols going around. The issue of misrecognitions is important; ideally this grammar would be given lower priority than the other grammars so that it would have the low priority that dictation usually has. Rather than just replacing words it might be more flexible to have a restricted set of commands available, especially for Lyx / scientific notebook where you need keystrokes not just text. I just posted David's suggestion on how to do this at the end of this issue dictation-toolbox/Caster#623 .

In general, this sort of thing would not be so much for a situation where you are doing heavy duty calculations but more for a situation with like 70-30 English-math symbols ratio or higher. I will be trying out this sort of thing in the wild myself so I will see how it goes. I think it will at least be good to have as an option which the user can easily enable/disable, but I will have a better sense after using for a while.

alexboche
@alexboche
adding printed forms many at a time is a little bit tricky. it appearss that the best method is to edit the XML file associated with thee vocabulary you are using. So far it seems like you might have to first export your vocabulary and save it as an XML file then edit the file and then reimport it because I cannot find where the XML file is stored within your user profile. Here is a sample XML file that shows how to add the printed forms for "math alpha" "math Alpha" and "math beta" to get $\alpha$ $\Alpha$ and $\beta$. https://gist.github.com/alexboche/c02af32574a40854d3ea2d4020a269f4
Let me know if anyone wants more info about that. Keeping the vocabulary in a .txt file is more compact then XML but does not support printed forms and thus does not support backslashes. In principle we can gennerate these XML files programmatically for all the combinations of letters and superscripts and so on.
Anna Kirkpatrick
@annakirkpatrick
@alexboche @mrob95 just some thoughts on the conversation above : While I have found it really helpful to add mathematical words and phrases to Dragon's vocabulary, I don't really see the value in trying to put those words into math fly itself (that is, as a new grammar). I understand what has been said above about the need to pause before and after dictating notation. I actually expected this to be really annoying when I first started using math fly, but I haven't found that to be the case. I've now written nine pages of math with Latex using math fly, and I haven't found the pauses to be particularly annoying. Since Latex already distinguishes text mode from math mode, it's pretty natural to do it in speech too. (I will also note that you would have to have an enormous grammar in order to include all of the words you might want to closely mix with math.)
Anna Kirkpatrick
@annakirkpatrick
A find and replace strategy was mentioned above. That is essentially the strategy that my previous system, NatLaTeX, used. It was slightly more complicated in that it did involve an actual parser that could, for example, distinguish math mode from text mode and apply different replacements in each context. So, the find and replace concept absolutely can work (and it eliminates all need to pause) but it also has some significant drawbacks. (The source is harder to read. Errors are harder to find. It's very easily for a slightly misrecognized word to be unnoticed when dictating but then not be replaced properly by the find and replace script. It can also make collaboration harder. It doesn't work well with text editors aimed at LaTeX. Keeping track of spacing/indentation is a nightmare.) I don't think mixing the grammar/command strategy used by math fly with the find and replace strategy is a good idea. I think you would end up with the limitations of both and quite a bit of confusion.
To address a couple other lingering points/questions:
-when it comes to computer algebra systems, I have mainly used sage (sagemath.org). I interacted with it mainly as plain text, sometimes in a Python-style interactive shell, sometimes writing scripts with a plain text editor and running from system shell or sage shell. There is a browser-based "GUI" but it mainly amounts to just another way to access the interactive shell. The ability to display graphs and typeset mathematics in the browser is nice, but otherwise it behaves like a shell.
-I totally forgot about printed forms! They are pain, but, yes, you can get backslashes that way.
I will briefly note that all of my comments above apply just to LaTeX. Using a GUI program like scientific notebook could be totally different. And there might be value in considering different strategies for these different contexts.
Mike Roberts
@mrob95

Pretty much agree. I think that the Dragon system of separating utterances into either dictation or commands is a fairly nice way of working, and it is easy to understand if it doesn't do what you expect. I would be hesitant to introduce a lot more complexity for the fairly minor benefit of reducing the number of pauses. I don't know about you but I pause quite a lot while I'm dictating anyway to think about what I need to say next and to make sure the previous section is correct.

One thing which may be of interest. There is a parallel voice control project called Talon which provides an API for defining grammars in the same way that Natlink + dragonfly does. Until now it has only been available for Mac but is currently being ported to Windows. One of the nice things which the Talon API exposes from Dragon is a parameter which controls how aggressive it is at finishing an utterance (how long of a pause it hears before it decides that you've finished talking). Reducing this would make the dictation -> pause -> commands cycle quicker.

Anna Kirkpatrick
@annakirkpatrick
@alexboche I finally got around to downloading some screen recording software and recording myself writing some code. https://youtu.be/BWwKV0W7d-k
alexboche
@alexboche
@annakirkpatrick Nice speed, that's impressive! Mind if I share that video on some other chat rooms?
alexboche
@alexboche
Nice you can talk while you're doing it
esc123
@esc123
@annakirkpatrick Awesome video. Great work on your keyboard layout as well!
Mike Roberts
@mrob95
Yeah very nice video. Interesting to see an alternative to dictation, I think I have probably been underestimating how good eye tracking is nowadays
quintijn
@quintijn
Hello Mike, I started with mathfly today, because I have to instruct a student who needs speech recognition and formula's. I must start by making my VERY BIG complements for this grammar. I also learned a bit more about dragonfly and caster, for me, stuck to Unimacro and Vocola, is in a lot of ways another way of thinking. Also thanks Alex Boche for putting me in the right direction.
I now have to see how to adapt for a Dutch user profile, as the student will be working in Dutch. To my pleasure the numbers behave well up-to-now! Making lots of translations in the config files should do most of the work. I am planning to use double words in many cases, so the user can also choose for the english variants when appropriate. The radio alphabet is happily the same.
quintijn
@quintijn
Mike, I tried to send you an email too, but the mail address on the instruction pages on your mathfly.org site seems to be not valid any more...
Mike Roberts
@mrob95

@quintijn good to hear :-) sorry about the email, it is redirecting to my old university email which is no longer valid, I'll update it.

Let me know if there is anything I can do, and if you want to make a pull request for the Dutch translations or just post them in here then that would be a good addition. I set it up so that most of the standard operations which a user might want to tweak are in the TOML configuration files, but that's obviously not particularly powerful for anything more than inserting characters or executing keystrokes. If anything more complex is required then that should be easily doable on the dragonfly side.

quintijn
@quintijn
@mrob95 has there ever been a thought about other user languages? Because I am working in 2 languages, that was always an essential part for me. All Vocola commands can be defined in each language, as you have to do it yourself. And all Unimacro grammars are "translatable". I think this will not be too easy for Dragonfly/Caster/Mathfly to realise I fear.
Mike Roberts
@mrob95
I haven't really thought about it, but I don't think it would be too difficult to manage. There would just need to be a setting in the settings file to switch language, or it could probably pull this from natlink. Then we just use a different set of config files, or maybe even a different set of files entirely.
Dragonfly has some support for multiple languages, but this is mainly limited to integers
quintijn
@quintijn
I was studying the lyx_nested rules, with integral etc. and I was wondering where to find the <sequence1>, <sequence2> etc. rules. Maybe you can help. Because the CCR is so important, and the _keystrokes.py grammar of Unimacro focusses on those, I was thinking about an effort to implement the same there. See how the one and the other performs.
@mrob95 well, integers are on the hardest to tackle! Unimacro has also quite complicated rules to catch numbers, and a quick try with your examples of 416 showed me "mine" seem to be better. It was original a grammar of Joel Gould in his first examples he gave, now nearly 20 years ago!!!!
For the symbols it could be good not to throw away all the english words, as a lot of things are possibly better memorised in english.
quintijn
@quintijn
"new math line | nieuwe math regel" = "escape:10, end, enter, c-m"
"squared| kwadraat"                         = "caret, 2, right"
"cubed|derde macht"                           = "caret, 3, right"
"inverse"                         = "caret, minus, 1, right"
"(parens | parentheses | haakjes)"          = "a-m, lparen"
"square brackets| brekkits | vierkante haakjes" = "a-m, lbracket"
"curly brackets | accolades"                  = "a-m, lbrace"
something like this. It is eventually up to the user to decide what words to use.
Mike Roberts
@mrob95

CCR is complicated, as you probably know, and I am more or less just hacking on top of caster's implementation.

There are some details on the mathfly site, but the the way it is all done is to merge everything that can be recognised in the current context into one rule, and then allow repetitions of that rule. The structure is something like

Rule(
    Repetition(
        Alternative(
            Compound("command one"->action)
            Compound("command two"->action)
            Compound("command three"->action)
        )
    )
)
That is the theory at least, the implementation is nowhere near as nice as I think it could be, and I'm working at the moment on completely re-implementing it.
quintijn
@quintijn
As another (general) example, I use "show messages" and "refresh messages" in both English and Dutch speech profiles. Dutch are then lucky that the languages are a bit similar. With French, Spanish or Italian it will get a lot more difficult.
Mike Roberts
@mrob95
Am I right in thinking that natlink can grab the current profile language?
quintijn
@quintijn
O yes! I look it up!
Mike Roberts
@mrob95
If so then at least a first implementation would just be to get the current language and redirect to e.g. "lyx_nl.toml"
quintijn
@quintijn
import natlinkstatus
language = natlinkstatus.getLanguage()
If would suggest to take nld for dutch as the language is returned so. If it is different from enx use the _nld etc. If it is enx (English) just keep it as it is.
Language in Dragon conventions is always a 3 letter code. In natlinkstatus.py (in the natlink core directory) are they all listed.(line 238)
Thanks Mike, I must quit here for the moment.
Mike Roberts
@mrob95
No worries, I guess you can keep going just adding alternatives for now? Might take me a couple of days to have a look at this
Mike Roberts
@mrob95
Very rough and I haven't done any testing with it, but theoretically at least you should be able to dump some new toml files in config/nld/ which will be imported if the language setting is equal to nld. English will always be used as a fallback
Mike Roberts
@mrob95

Major architectural change. Before updating you will need to run

pip install dfly-breathe

I have entirely replaced the old caster-based CCR backend with Breathe, a library I've been working on. Should make it much easier to add support for new apps etc, and I have fully fleshed out the NestedRule concept which was originally prototyped here. Please let me know if there are any issues.

esc123
@esc123
Cool, Ill give it a test run in the next few days
LexiconCode
@LexiconCode
Were is 'NestedRule' used?
Mike Roberts
@mrob95

Well basically I wanted rules that were like:

Integral from <stuff1> to <stuff2>

Where stuff was a sequence of any other ccr commands. Nested rules were the first implementation based on Caster, breathe allows this by allowing the user to add custom top-level rules. The mathfly examples are in e.g. lyx.py at the bottom but may be slightly difficult to understand at a glance, there are clearer examples on the breathe readme which show greedy dictation and optional repeat values using the same feature.

You just have to define a CommandsRef extra which references a sequence of commands and then include an Exec action to execute the sequence
LexiconCode
@LexiconCode
Awesome I look forward to tinker with mathfly in next few weeks :)
Mike Roberts
@mrob95
Greedy dictation was a feature I really wanted in Breathe, the fact that I could do this as well using the same feature is a nice bonus ;)