Hi @/all Long hours of pondering the code I have decided to rewrite the whole state machine in something that is not a state machine, see main/state/susi-state-machine.py-simple. There have been several problems:
- async changes of state due to timeouts
- recursion limits in theory due to iterated function calls exhausting the call stack
The new code is now really simple: start recogize, and the callback does all. All in all a few lines of code. The only thing not transferred over by now is the scheduler for delayed actions, since by now I don't know how this works, need to look into it.
All in all the code is much simpler, easier to read IMHO, one single file, and works at least as good as the current version. To give it a test, just rename it to susi-state-machine.py and it should work.
@hongquan @stealthanthrax concerning susi_linux: I am contemplating dropping the whole state machine stuff. The current code in development branch already uses the susi_state_machine_simple.py code which contains all in one file, does not have call stack problems, properly deals with threads, fixes several problems with the current code, etc.
Do you have any objections against this move?
@norbusan , how will it handle a second query? e.g. Getting the stop command while music is playing?
I am contemplating dropping the whole state machine stuff. The current code in development branch already uses the susi_state_machine_simple.py code which contains all in one file, does not have call stack problems, properly deals with threads, fixes several problems with the current code, etc.
I already mentioned that: there are two ways of reactions: one is "say" which is used for answering simple quieries (what is ..., who is ..), and during these playbacks no interuption is possible. But playing music is started in the background and the loop returns immediately to the normal processing mode, so stop commands etc are already all working.
This is the same status as of what we had before.
start which listens for the keyword, and if it is found, jumps into the keyword callback. From there the STT and susi server connection is done, and answers processed. After this one returns to the start function which does a while True
Hello all mentors and contributors, I wanted to contribute to this project since I have significant experience working on Linux projects (like meilix) before and wanted to get involved with this project. So I started going through the projects and found that they are many repositories related to susi_linux.
There are some questions that I wanted to ask moving forward,
What are some elementary steps working towards the project?
I have successfully installed susi_linux desktop client on my workstation using susi_installer and found some issues that were faced by me and some students participating in Google Code-In. I would like to work on these issues.
Are we following the same approach for raspi because I couldn't find a separate section for installation in raspi? (Or a separate disk image for raspi).
What are some issues that we are currently facing?
I would be very happy if I could help in solving some specific issues that we are currently facing in the project.
Report on RPi4 with DeepSpeech 0.6.0: abysmal - extremely slow, detection is like, well, every third word might be correct ... so definitely a no go for now...
I have worked with Deepspeech stt on raspi and we found that it is much better than other stt clients that we tested for our patent.
What are the plans for the project’s future?
@mariobehling told me that we are working on building SUSI.AI Linux distribution (something like meilix). What is the progress of that idea? I see many issues here https://github.com/fossasia/labs.fossasia.org/labels/SUSI%20AI that are open for susi_linux. Which issue are we currently working on? What issues are we targeting for this year’s summer?
@pranav1698 thanks for your interest. Concerning the issues, please open an issue at the susi_linux project for issues related to the actual client, and susi_installer for issues related to the installation and RPi setup.
If you want to see how we build RPi images, take a look at .travis/create-image script in susi_installer, it is more or less mounting the base image, installing prerequisites, and then running the susi installer with the necessary options.
Concerning your deepspeech experience, I am very interested to see more details. I have added deepspeech support to the python speech_recognition library, forked in the fossasia github org, and tried it on my RPi4, and unfortunately detection of words was really bad. I would love to have someone with more knowledge about deepspeech/rpi. On the desktop it works quite nice.
Concerning the future of SUSI.AI: I think at the moment there are two big open problems: The susi server has some issues, in particular on the RPi at the moment, which holds back new releases. The other is integration with desktop environments. And finally, good skills.
I am doing most of the susi_linux and susi_installer work, so feel free to ping me whenever you have a questions.
Also, we used deepspeech on rpi3. I cannot share the script that I wrote for using deepspeech due to copyright issues, but it was working well as compared to other similar clients like
pocketsphinx when we tested it for our use-case. I personally feel that if we could run Google/IBM stt modules locally then the output would be much better but there were time limits on our previous project so we were not able to move forward on that idea.
recognize_deepspeech function I added to speech_recognition: https://github.com/fossasia/speech_recognition/blob/e452e9f3295232a6f5de0dc789acf2e1a4311f5c/speech_recognition/__init__.py#L846 and let me know whether there is anything that needs to be improved?
speech_recognition does that, and susi supports selecting google/watson/...)
@pranav1698 it would be great if you could look into the
recognize_deepspeechfunction I added tospeech_recognition: https://github.com/fossasia/speech_recognition/blob/e452e9f3295232a6f5de0dc789acf2e1a4311f5c/speech_recognition/__init__.py#L846 and let me know whether there is anything that needs to be improved?
Sure
Just for own memory, I finally understand why we got repeated hotword detections: The callback was only a rx next subject, thus immediately returned. This events were all queued by rx, and delivered after the initial callback (subscribe) was finished. Thus, occurrences of "susi" in the answer triggered new reactions via hotword detection.
Many of the problems appeared due to the async planned actions callbacks, but as well as via the wake button callbacks.
I think the proper solution is also not the _simple.py stuff I was doing, because it requires stopping and restarting the detection, which is suboptimal from efficiency POV. What we need is an actor model (pykka) where the hotword detection actor sends a message to the STT actor etc etc, and everything works async. That way we can deal with planned actions just as another message in the queue.
Maybe there are other solutions, I am thinking, but this is what I currently plan.
MOre details: the problem is with PyAUdio that is used in callback mode in a separate thread, so "susi" calls during processing are still recorded in snowboy_detector (into the ringbuffer), and thus processed after the current callback terminates.
This means, we either need to rewrite the snowboy_detector code to not use pyaudio in callback mode, or do something else ...
I am contemplating simply waiting for say 0.3s after returning from the callback and ignore all hotword detections in this time. Not optimal. Thinking ...
For the record, I have now fixed it by waiting 0.05s (that seems to be enough) after the return before putting the device into idle mode again. This way we can have the recognizer run permanently and still ignore susi words during normal answers.
If we ever change to a more complicated and 3-syllable hotword, this should be not necessary I guess.
@/all I have tagged a new release on github https://github.com/fossasia/susi_installer/releases/tag/build-20200108.1
This release works quite well for me on my RPi4. Besides having fixed server, fixed client, there is also a new feature: one can change the hotword my
~/SUSI.AI/bin/susi-config set hotword.model=computer.pmdl
sudo systemctl restart ss-susi-linux@piThis would switch to the hotword "computer". Other models are Attention.pmdl, Robot.pmdl, jarvis.pmdl, and the default susi.pmdl
Please give it a try, both on the RPi and on the Desktop, but use the development branches on the desktop.
The event queue is only used for the planned actions, not for anything else. That way we don't need async calls anywhere.
@norbusan Right now it is only possible to set "Planned Actions". It is not possible to unschedule/delete them. It should be possible to "cancel all alarms", "cancel a planned action tomorrow" (but keep others) etc. - just like in a calendar. What are your ideas to achieve this?
@mariobehling First of all, planned actions are anyway broken at the moment, either because the server answer changed, or the implementation was broken from the beginning. I realized that while rewriting the engine. See this comment in the source code
# TODO TODO
# plan_delay is wrong, it is 0, we need to use
# plan = {'language': 'en', 'answer': 'ALARM', 'plan_delay': 0,
# 'plan_date': '2019-12-30T13:36:05.458Z'}
# plan_date !!!!!
self.action_schduler.add_event(int(plan['plan_delay']) / 1000, plan)Then, concerning cancelling planned actions. I think this can be done, but we need:
- a skill/server support so that the server answers properly
- maybe types of planned actions to be deleted?
Then I can implement this easily by going through the Queue used for planned events and drop those that are to be cancelled.
@mariobehling it seems that my TODO there is actually wrong, and the code works as is, but the answers of the susi server are strange:
>>> import susi_python as susi
local server is down
>>> susi.ask("Set alarm for 5 minutes")
{'planned_actions': [{'language': 'en', 'answer': 'ALARM', 'plan_delay': 0, 'plan_date': '2020-01-09T01:59:53.199Z'}], 'language': 'en', 'answer': 'alarm set for for 5 minutes'}
>>> susi.ask("Set alarm in 5 minutes")
{'planned_actions': [{'language': 'en', 'answer': 'ALARM', 'plan_delay': 300001, 'plan_date': '2020-01-09T02:05:10.377Z'}], 'language': 'en', 'answer': 'alarm set for in 5 minutes'}
>>>I get planned actions in both cases, but one with 0 delay that is immediately. Maybe this is only a question of skill development ...
@hongquan @stealthanthrax I have started to rework the layout of susi_linux into a more "standard" form:
- the main module is not
mainbutsusi_linux - modules with only one file aren't put into sub-directories
- removal of unused stuff
- drop traces of state machine, rename
susi_state_machine_simple.pytosusi_loop - ...
The branch is here: https://github.com/norbusan/susi_linux/tree/layout-rework
What do you think about this?
@mariobehling I opened a PR fossasia/susi_linux#531 would be nice if someone can review it
@pranav1698 what do you mean concerning "work on that issue"? The branch is already prepared ;-) There can still be more things be done:
- adding a
setup.pywith distutils integration - better dealing with config file
config.json- I would like to move it to ~/.susi.config or similar instead of the CWD - after the previous step, rework the scripts and wrappers, so the wrappers become unnecessary
- ...