For this choice, if "last" is spoken, would the returned value be "last | nine" or "last"? I'm not 100% sure offhand, so it would probably be a good idea for the documentation to make it clear. 😃

@Danesprite It seems to build and look fine to me. Am I missing something?

Here are installation instructions for anyone interested in testing this out:

1. Install Talon.
2. Clone dragonfly (or use existing clone): git clone [email protected]:dictation-toolbox/dragonfly.git dragonfly && cd dragonfly
3. Check out this pull request: git fetch origin pull/326/head:talon && git checkout talon
4. Pip install your dragonfly clone into Talon's copy of Python, along with any dependencies of your dragonfly grammars:

cd C:\Program Files\Talon
python -m pip install -e c:\path\to\dragonfly\clone
python -m pip install <other modules you need for your grammars>

5. Copy your dragonfly grammars into your Talon user directory (on Windows: C:\Users\<username>\AppData\Roaming\talon\user). Feel free to nest these in a subdirectory.
6. Start Talon and check the log for any issues (right-click Talon > Scripting > View Log).
7. Start using the commands in your Dragonfly grammars! They'll work with any Talon speech backend (Dragon, Conformer, etc).

@Danesprite I think that makes sense and would be a good change. Also, I wonder if there should be a conspicuous warning message or something when this happens, where get_engine is called automatically because a grammar is loaded or whatever? This has definitely occurred other times, and been a bit confusing.

I'm not sure if this is an issue or I'm making a mistake somewhere, but every time I seem to import Grammar in a file and create functions to load them in the main.py, the engine changes Kaldi to sapi5 and I also get this error that Natlink requires 32bit python. Then no speech is detected at all. However if I create, add and load the grammars directly in main.py it works.

main.py is the same as kaldi_module_loader_plus.py except for the importing loadL1 and unloadL1 and using them when the system is awake and sleeping

I also found this issue occurring when I use Integer or IntegerRef.

# main.py
"""
Command-module loader for Kaldi.

This script is based on 'dfly-loader-wsr.py' written by Christo Butcher and
has been adapted to work with the Kaldi engine instead.

This script can be used to look for Dragonfly command-modules for use with
the Kaldi engine. It scans the directory it's in and loads any ``_*.py`` it
finds.
"""


# TODO Have a simple GUI for pausing, resuming, cancelling and stopping
# recognition, etc

from __future__ import print_function

import logging
import os.path
import sys

import six

from dragonfly import get_engine, get_current_engine
from dragonfly import Grammar, MappingRule, Function, Dictation, FuncContext
from dragonfly.loader import CommandModuleDirectory
from dragonfly.log import setup_log
# from CustomGrammars import l1Grammars
from grammarContoller import loadL1, unloadL1


# --------------------------------------------------------------------------
# Set up basic logging.

if False:
    # Debugging logging for reporting trouble
    logging.basicConfig(level=10)
    logging.getLogger('grammar.decode').setLevel(20)
    logging.getLogger('grammar.begin').setLevel(20)
    logging.getLogger('compound').setLevel(20)
    logging.getLogger('kaldi.compiler').setLevel(10)
else:
    setup_log()


# --------------------------------------------------------------------------
# User notification / rudimentary UI. MODIFY AS DESIRED

# For message in ('sleep', 'wake')
def notify(message):
    if message == 'sleep':
        print("Sleeping...")
        # get_engine().speak("Sleeping")
    elif message == 'wake':
        print("Awake...")
        # get_engine().speak("Awake")


# --------------------------------------------------------------------------
# Sleep/wake grammar. (This can be unused or removed if you don't want it.)

sleeping = False

def load_sleep_wake_grammar(initial_awake):
    sleep_grammar = Grammar("sleep")
    def sleep(force=False):
        global sleeping
        if not sleeping or force:
            sleeping = True
            unloadL1()
            sleep_grammar.set_exclusiveness(True)
        notify('sleep')

    def wake(force=False):
        global sleeping
        if sleeping or force:
            sleeping = False
            loadL1()
            sleep_grammar.set_exclusiveness(False)
        notify('wake')

    class SleepRule(MappingRule):
        mapping = {
            "start listening":  Function(wake) + Function(lambda: get_engine().start_saving_adaptation_state()),
            "stop listening":   Function(lambda: get_engine().stop_saving_adaptation_state()) + Function(sleep),
            "halt listening":   Function(lambda: get_engine().stop_saving_adaptation_state()) + Function(sleep),
        }
    sleep_grammar.add_rule(SleepRule())

    sleep_noise_rule = MappingRule(
        name = "sleep_noise_rule",
        mapping = { "<text>": Function(lambda text: False and print(text)) },
        extras = [ Dictation("text") ],
        context = FuncContext(lambda: sleeping),
    )
    sleep_grammar.add_rule(sleep_noise_rule)

    sleep_grammar.load()

    if initial_awake:
        wake(force=True)
    else:
        sleep(force=True)


# --------------------------------------------------------------------------
# Main event driving loop.

def main():
    logging.basicConfig(level=logging.INFO)

    try:
        path = os.path.dirname(__file__)
    except NameError:
        # The "__file__" name is not always available, for example
        # when this module is run from PythonWin.  In this case we
        # simply use the current working directory.
        path = os.getcwd()
        __file__ = os.path.join(path, "kaldi_module_loader_plus.py")

    # Set any configuration options here as keyword arguments.
    # See Kaldi engine documentation for all available options and more info.
    engine = get_engine('kaldi',
        model_dir='../kaldi_model',  # default model directory
        # vad_aggressiveness=3,  # default aggressiveness of VAD
        # vad_padding_start_ms=150,  # default ms of required silence before VAD
        # vad_padding_end_ms=150,  # default ms of required silence after VAD
        # vad_complex_padding_end_ms=500,  # default ms of required silence after VAD for complex utterances
        # input_device_index=None,  # set to an int to choose a non-default microphone
        lazy_compilation=True,  # set to True to parallelize & speed up loading
        # retain_dir=None,  # set to a writable directory path to retain recognition metadata and/or audio data
        # retain_audio=None,  # set to True to retain speech data wave files in the retain_dir (if set)
    )

    # Call connect() now that the engine configuration is set.
    engine.connect()

    # Load grammars.
    load_sleep_wake_grammar(True)

    directory = CommandModuleDirectory(path, excludes=[__file__])
    directory.load()

    # Define recognition callback functions.
    def on_begin():
        print("Speech start detected.")

    def on_recognition(words):
        message = u"Recognized: %s" % u" ".join(words)

        # This only seems to be an issue with Python 2.7 on Windows.
        if six.PY2:
            encoding = sys.stdout.encoding or "ascii"
            message = message.encode(encoding, errors='replace')
        print(message)

    def on_failure():
        print("Sorry, what was that?")

    # Start the engine's main recognition loop
    engine.prepare_for_recognition()
    try:
        print("Listening...")
        print(get_current_engine())
        engine.do_recognition(on_begin, on_recognition, on_failure)
    except KeyboardInterrupt:
        pass


if __name__ == "__main__":
    main()

# grammarContoller.py File

from dragonfly import Grammar
from CustomGrammars import l1Grammars


l1_grammar = Grammar('Level 1 Grammar')
l1_grammar.add_rule(l1Grammars.MainGrammarRules())
l1_grammar.add_rule(l1Grammars.FunctionGrammars())

def unloadL1():
    l1_grammar.unload()

def loadL1():
    l1_grammar.load()

Thanks @daanzu Solved the problem

I haven't tested this myself to verify it, but I think your problem may be that you are importing and building the grammars before you have initially called get_engine. What if you move the import line to be after that? By the way, you don't actually need to load and unload your grammars for the wake/sleep: that grammar is specially marked as exclusive, so it should automatically effectively disable all other grammars (that are not also marked as exclusive) whenever it is asleep.

Seems to have solved the problem. I shall test it a little more and close the issue

I haven't tested this myself to verify it, but I think your problem may be that you are importing and building the grammars before you have initially called get_engine. What if you move the import line to be after that? By the way, you don't actually need to load and unload your grammars for the wake/sleep: that grammar is specially marked as exclusive, so it should automatically effectively disable all other grammars (that are not also marked as exclusive) whenever it is asleep.

I'm not sure if this is an issue or I'm making a mistake somewhere, but every time I seem to import Grammar in a file and create functions to load them in the main.py, the engine changes Kaldi to sapi5 and I also get this error that Natlink requires 32bit python. Then no speech is detected at all. However if I create, add and load the grammars directly in main.py it works.

main.py is the same as kaldi_module_loader_plus.py except for the importing loadL1 and unloadL1 and using them when the system is awake and sleeping

I also found this issue occurring when I use Integer or IntegerRef.

# main.py
"""
Command-module loader for Kaldi.

This script is based on 'dfly-loader-wsr.py' written by Christo Butcher and
has been adapted to work with the Kaldi engine instead.

This script can be used to look for Dragonfly command-modules for use with
the Kaldi engine. It scans the directory it's in and loads any ``_*.py`` it
finds.
"""


# TODO Have a simple GUI for pausing, resuming, cancelling and stopping
# recognition, etc

from __future__ import print_function

import logging
import os.path
import sys

import six

from dragonfly import get_engine, get_current_engine
from dragonfly import Grammar, MappingRule, Function, Dictation, FuncContext
from dragonfly.loader import CommandModuleDirectory
from dragonfly.log import setup_log
# from CustomGrammars import l1Grammars
from grammarContoller import loadL1, unloadL1


# --------------------------------------------------------------------------
# Set up basic logging.

if False:
    # Debugging logging for reporting trouble
    logging.basicConfig(level=10)
    logging.getLogger('grammar.decode').setLevel(20)
    logging.getLogger('grammar.begin').setLevel(20)
    logging.getLogger('compound').setLevel(20)
    logging.getLogger('kaldi.compiler').setLevel(10)
else:
    setup_log()


# --------------------------------------------------------------------------
# User notification / rudimentary UI. MODIFY AS DESIRED

# For message in ('sleep', 'wake')
def notify(message):
    if message == 'sleep':
        print("Sleeping...")
        # get_engine().speak("Sleeping")
    elif message == 'wake':
        print("Awake...")
        # get_engine().speak("Awake")


# --------------------------------------------------------------------------
# Sleep/wake grammar. (This can be unused or removed if you don't want it.)

sleeping = False

def load_sleep_wake_grammar(initial_awake):
    sleep_grammar = Grammar("sleep")
    def sleep(force=False):
        global sleeping
        if not sleeping or force:
            sleeping = True
            unloadL1()
            sleep_grammar.set_exclusiveness(True)
        notify('sleep')

    def wake(force=False):
        global sleeping
        if sleeping or force:
            sleeping = False
            loadL1()
            sleep_grammar.set_exclusiveness(False)
        notify('wake')

    class SleepRule(MappingRule):
        mapping = {
            "start listening":  Function(wake) + Function(lambda: get_engine().start_saving_adaptation_state()),
            "stop listening":   Function(lambda: get_engine().stop_saving_adaptation_state()) + Function(sleep),
            "halt listening":   Function(lambda: get_engine().stop_saving_adaptation_state()) + Function(sleep),
        }
    sleep_grammar.add_rule(SleepRule())

    sleep_noise_rule = MappingRule(
        name = "sleep_noise_rule",
        mapping = { "<text>": Function(lambda text: False and print(text)) },
        extras = [ Dictation("text") ],
        context = FuncContext(lambda: sleeping),
    )
    sleep_grammar.add_rule(sleep_noise_rule)

    sleep_grammar.load()

    if initial_awake:
        wake(force=True)
    else:
        sleep(force=True)


# --------------------------------------------------------------------------
# Main event driving loop.

def main():
    logging.basicConfig(level=logging.INFO)

    try:
        path = os.path.dirname(__file__)
    except NameError:
        # The "__file__" name is not always available, for example
        # when this module is run from PythonWin.  In this case we
        # simply use the current working directory.
        path = os.getcwd()
        __file__ = os.path.join(path, "kaldi_module_loader_plus.py")

    # Set any configuration options here as keyword arguments.
    # See Kaldi engine documentation for all available options and more info.
    engine = get_engine('kaldi',
        model_dir='../kaldi_model',  # default model directory
        # vad_aggressiveness=3,  # default aggressiveness of VAD
        # vad_padding_start_ms=150,  # default ms of required silence before VAD
        # vad_padding_end_ms=150,  # default ms of required silence after VAD
        # vad_complex_padding_end_ms=500,  # default ms of required silence after VAD for complex utterances
        # input_device_index=None,  # set to an int to choose a non-default microphone
        lazy_compilation=True,  # set to True to parallelize & speed up loading
        # retain_dir=None,  # set to a writable directory path to retain recognition metadata and/or audio data
        # retain_audio=None,  # set to True to retain speech data wave files in the retain_dir (if set)
    )

    # Call connect() now that the engine configuration is set.
    engine.connect()

    # Load grammars.
    load_sleep_wake_grammar(True)

    directory = CommandModuleDirectory(path, excludes=[__file__])
    directory.load()

    # Define recognition callback functions.
    def on_begin():
        print("Speech start detected.")

    def on_recognition(words):
        message = u"Recognized: %s" % u" ".join(words)

        # This only seems to be an issue with Python 2.7 on Windows.
        if six.PY2:
            encoding = sys.stdout.encoding or "ascii"
            message = message.encode(encoding, errors='replace')
        print(message)

    def on_failure():
        print("Sorry, what was that?")

    # Start the engine's main recognition loop
    engine.prepare_for_recognition()
    try:
        print("Listening...")
        print(get_current_engine())
        engine.do_recognition(on_begin, on_recognition, on_failure)
    except KeyboardInterrupt:
        pass


if __name__ == "__main__":
    main()

# grammarContoller.py File

from dragonfly import Grammar
from CustomGrammars import l1Grammars


l1_grammar = Grammar('Level 1 Grammar')
l1_grammar.add_rule(l1Grammars.MainGrammarRules())
l1_grammar.add_rule(l1Grammars.FunctionGrammars())

def unloadL1():
    l1_grammar.unload()

def loadL1():
    l1_grammar.load()

Does dragonfly normally allow two grammars to have the same name?

Sorry, bad test case. If I change the second name, it does work properly. But there is a bug to report -- it just takes a more complicated example to demonstrate:

from dragonfly import (
    Dictation,
    Grammar,
    IntegerRef,
    Key,
    MappingRule,
    Text,
)

class CommandRule(MappingRule):
    mapping = {
        "hello world [<n>]": Text("This is working!"),
    }
    extras = [IntegerRef("n", 1, 10, default=1)]

grammar = Grammar("DragonflyTest")
grammar.add_rule(CommandRule())
grammar.load()


class SecondCommandRule(MappingRule):
    mapping = {
        "goodbye world [<n>]": Text("This is also working!"),
    }
    extras = [IntegerRef("n", 1, 10, default=1)]

second_grammar = Grammar("DragonflyTest2")
second_grammar.add_rule(SecondCommandRule())
second_grammar.load()

Note multiple conditions for this repro to work (I tested without each of these):

1. IntegerRef must be added and used in the grammars.
2. Two grammars using IntegerRef must be present.

Fails like so:

2021-04-11 18:03:01 ERROR defer_load() failed
   18:                                                                      threading.py:912* # loader thread
   17:                                                                      threading.py:954* 
   16:                                                                      threading.py:892* 
   15:                                                           app\resources\loader.py:715| 
   14:                                                           app\resources\loader.py:693| 
   13:                                                           app\resources\loader.py:235| 
   12:                                                  user\dragonfly\dragonfly_test.py:29 | second_grammar.load()
   11:         c:\users\james\projects\talon_dragonfly\dragonfly\grammar\grammar_base.py:451| lst._update()
   10: c:\users\james\projects\talon_dragonfly\dragonfly\engines\backend_talon\engine.py:67 | self._grammar_wrappers[grammar.name] =..
    9: c:\users\james\projects\talon_dragonfly\dragonfly\engines\backend_talon\engine.py:82 | return wrapper
    8:                                                   talon\experimental\dragonfly.py:87 | 
    7:                                                         talon\scripting\module.py:80 | 
    6:                                                           talon\scripting\rctx.py:42 | 
    5:                                                           talon\scripting\rctx.py:96 | # [stack splice]
    4:                                                           talon\scripting\rctx.py:59 | 
    3:                                                                          <string>:1  | 
    2:                                                         talon\scripting\module.py:82 | 
    1:                                                       talon\scripting\registry.py:188| 
talon.scripting.types.DuplicateError: user.dragonfly.dragonfly_test
2021-04-11 18:03:01    IO root (ERROR): defer_load() failed
   18:                                                                      threading.py:912* # loader thread
   17:                                                                      threading.py:954* 
   16:                                                                      threading.py:892* 
   15:                                                           app\resources\loader.py:715| 
   14:                                                           app\resources\loader.py:693| 
   13:                                                           app\resources\loader.py:235| 
   12:                                                  user\dragonfly\dragonfly_test.py:29 | second_grammar.load()
   11:         c:\users\james\projects\talon_dragonfly\dragonfly\grammar\grammar_base.py:451| lst._update()
   10: c:\users\james\projects\talon_dragonfly\dragonfly\engines\backend_talon\engine.py:67 | self._grammar_wrappers[grammar.name] =..
    9: c:\users\james\projects\talon_dragonfly\dragonfly\engines\backend_talon\engine.py:82 | return wrapper
    8:                                                   talon\experimental\dragonfly.py:87 | 
    7:                                                         talon\scripting\module.py:80 | 
    6:                                                           talon\scripting\rctx.py:42 | 
    5:                                                           talon\scripting\rctx.py:96 | # [stack splice]
    4:                                                           talon\scripting\rctx.py:59 | 
    3:                                                                          <string>:1  | 
    2:                                                         talon\scripting\module.py:82 | 
    1:                                                       talon\scripting\registry.py:188| 
talon.scripting.types.DuplicateError: user.dragonfly.dragonfly_test
2021-04-11 18:03:02 WARNING rule referenced but not declared: user.dragonfly_DragonflyTest2__IntegerRef_36
2021-04-11 18:03:02    IO root (WARNING): rule referenced but not declared: user.dragonfly_DragonflyTest2__IntegerRef_36
2021-04-11 18:03:02 WARNING rule referenced but not declared: user.dragonfly_DragonflyTest2__IntegerRef_36
2021-04-11 18:03:02    IO root (WARNING): rule referenced but not declared: user.dragonfly_DragonflyTest2__IntegerRef_36

Does dragonfly normally allow two grammars to have the same name?

Also, regarding scipy: I'm currently using scikit-image which pulls this in. I am mostly just using this for Otsu thresholding. The way the code is structured, thresholding can be applied independently of the OCR backend, but in practice it is not needed for WinRT (just Tesseract). OpenCV also supports Otsu thresholding although it does not support adaptive Otsu thresholding out of the box, which I use in my "quality" presets for constructing a Tesseract reader. So basically, I could probably make it an optional dependency but it would be a minor pain to structure and I probably won't get to it right away. How do you feel about the other dependencies:

        "numpy",
        "pillow",
        "rapidfuzz",

Here's a simple example that demonstrates multiple grammars in one file not working:

from dragonfly import (
    Dictation,
    Grammar,
    IntegerRef,
    Key,
    MappingRule,
    Text,
)

class CommandRule(MappingRule):
    mapping = {
        "hello world": Text("This is working!"),
    }

grammar = Grammar("DragonflyTest")
grammar.add_rule(CommandRule())
grammar.load()


class SecondCommandRule(MappingRule):
    mapping = {
        "goodbye world": Text("This is also working!"),
    }

second_grammar = Grammar("DragonflyTest")
second_grammar.add_rule(SecondCommandRule())
second_grammar.load()

Fails with:

2021-04-11 13:04:58 ERROR defer_load() failed
   23:                                                                      threading.py:912* # cron thread
   22:                                                                      threading.py:954* 
   21:                                                                      threading.py:892* 
   20:                                                                     talon\cron.py:123| 
   19:                                                                     talon\cron.py:83 | 
   18:                                                                       talon\fs.py:54 | 
   17:                                                                       talon\fs.py:47 | 
   16:                                                           talon\scripting\rctx.py:225| # 'fs' main:on_change()
   15:                                                           app\resources\loader.py:669| 
   14:                                                           app\resources\loader.py:621| 
   13:                                                           app\resources\loader.py:235| 
   12:                                                  user\dragonfly\dragonfly_test.py:27 | second_grammar.load()
   11:         c:\users\james\projects\talon_dragonfly\dragonfly\grammar\grammar_base.py:451| lst._update()
   10: c:\users\james\projects\talon_dragonfly\dragonfly\engines\backend_talon\engine.py:67 | self._grammar_wrappers[grammar.name] =..
    9: c:\users\james\projects\talon_dragonfly\dragonfly\engines\backend_talon\engine.py:82 | return wrapper
    8:                                                   talon\experimental\dragonfly.py:87 | 
    7:                                                        talon\scripting\context.py:282| 
    6:                                                           talon\scripting\rctx.py:42 | 
    5:                                                           talon\scripting\rctx.py:96 | # [stack splice]
    4:                                                           talon\scripting\rctx.py:59 | 
    3:                                                                          <string>:1  | 
    2:                                                        talon\scripting\context.py:290| 
    1:                                                       talon\scripting\registry.py:267| 
talon.scripting.types.DuplicateError: user.dragonfly.dragonfly_test.DragonflyTest

Sounds good.

Not urgent to get this working on my machine, but I'll note that the latest published beta (talon-windows-108-0.1.5-51-g3121.exe) still has this bug.

Cool, can you come up with a repro for grammars not working so I can work on it?

For scipy, is it possible for you to remove that as a dependency? We do have opencv and numpy available in Talon at the moment.

The python executable in the Talon directory is pure cpython, but the python running inside Talon is not exactly. Complex dependencies like scipy will work best if I ship them with Talon (they won't import on Mac otherwise, for example). I'm not currently shipping scipy because it's a bit heavy.

I would probably just use id(screen) there

That's an unrelated bug in Talon on Windows. The fix is already in beta and will be released as part of v0.2. As a workaround, you should be able to do:

if flag: ctrl.mouse_click(button, down=True)
else: ctrl.mouse_click(button, up=True)

There appears to be a bug somewhere here such that Mouse("left").execute() clicks down but doesn't release the click (the expected behavior without an "up" or "down" qualifier).

This throws an error that screen is unhashable. You need to pass in a unique integer here (I got it working with hash(screen.name) but there are probably better choices).

Sorry, I forgot this account.

It is Android 10. German voice is installed.

Yes, I think this should be ready for merging now, as far as I know, unless any other testers pipe up.

Re: testing with audio. True, for this, slowing of the processing down to real time is the code path that requires extra logic. The Kaldi backend supports both methods.

Re: custom words. They should be added automatically with estimated pronunciations. Is this not working for you?