Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 25 10:36
    norbusan commented #1417
  • Jan 25 09:57
    Dilshaad21 commented #1417
  • Jan 23 20:13
    pranavkarthik10 closed #1429
  • Jan 21 15:48
    satyamohlan opened #1443
  • Jan 21 11:48
    parikshitgupta1 commented #1435
  • Jan 21 11:46
    norbusan commented #1435
  • Jan 21 11:45
    parikshitgupta1 commented #1428
  • Jan 21 11:40
    parikshitgupta1 commented #1435
  • Jan 21 11:04
    boy-who-codes opened #1442
  • Jan 20 15:56
    snitin315 commented #533
  • Jan 20 15:56
    snitin315 commented #533
  • Jan 20 15:08
    snitin315 commented #533
  • Jan 20 15:08
    snitin315 commented #533
  • Jan 20 10:52

    norbusan on release-20200120.0

    (compare)

  • Jan 20 10:52

    norbusan on master

    fix bug in travis yml (compare)

  • Jan 20 10:50

    norbusan on build-20200120.1

    (compare)

  • Jan 20 10:49

    norbusan on release-20200120.0

    (compare)

  • Jan 20 08:40

    norbusan on release-20200120.0

    (compare)

  • Jan 20 08:37

    norbusan on release-20200120.0

    (compare)

  • Jan 20 08:37

    norbusan on master

    Update README.md Update README.md Merge branch 'development' (compare)

Norbert Preining
@norbusan
@Orbiter yes, it listens during music playback, but not during normal answering.
I tried it, it worked here.
Mario Behling
@mariobehling
:thumbsup:
Norbert Preining
@norbusan
@/all in case you want to see the installation and actual running on Ubuntu 18.04, here is a transcript I made https://pastebin.com/tkzdtpk1
Norbert Preining
@norbusan

Hi @/all Long hours of pondering the code I have decided to rewrite the whole state machine in something that is not a state machine, see main/state/susi-state-machine.py-simple. There have been several problems:

  • async changes of state due to timeouts
  • recursion limits in theory due to iterated function calls exhausting the call stack

The new code is now really simple: start recogize, and the callback does all. All in all a few lines of code. The only thing not transferred over by now is the scheduler for delayed actions, since by now I don't know how this works, need to look into it.

All in all the code is much simpler, easier to read IMHO, one single file, and works at least as good as the current version. To give it a test, just rename it to susi-state-machine.py and it should work.

Norbert Preining
@norbusan
scheduler should work now, too, but there is a bug in the code (in both the current running code) that the wrong information is evaluated and the scedule not done properly. Grrr...
Norbert Preining
@norbusan
I have switched susi_linux now over to the new system in susi_state_machine_simple.py - there are already several bugs fixed that are in the original code and not easy to fix there (timeout not in separate thread, etc)
Norbert Preining
@norbusan

@hongquan @stealthanthrax concerning susi_linux: I am contemplating dropping the whole state machine stuff. The current code in development branch already uses the susi_state_machine_simple.py code which contains all in one file, does not have call stack problems, properly deals with threads, fixes several problems with the current code, etc.

Do you have any objections against this move?

Sanskar Jethi
@stealthanthrax

@norbusan , how will it handle a second query? e.g. Getting the stop command while music is playing?

I am contemplating dropping the whole state machine stuff. The current code in development branch already uses the susi_state_machine_simple.py code which contains all in one file, does not have call stack problems, properly deals with threads, fixes several problems with the current code, etc.

Norbert Preining
@norbusan

I already mentioned that: there are two ways of reactions: one is "say" which is used for answering simple quieries (what is ..., who is ..), and during these playbacks no interuption is possible. But playing music is started in the background and the loop returns immediately to the normal processing mode, so stop commands etc are already all working.

This is the same status as of what we had before.

THe state machine approach wasn't better in this respect, it only messed around with threads and async reentrance while other procedures were still running.
It works, many times tested. Only on the raspi the susi server is completely hosed, that is why we cannot do anything atm.
Sanskar Jethi
@stealthanthrax

I understand.

I already mentioned that: there are two ways of reactions: one is "say" which is used for answering simple quieries (what is ..., who is ..), and during these playbacks no interuption is possible.

Norbert Preining
@norbusan
Please look into the new susi_state_machine_simple.py, it is really rather simple ...
Sanskar Jethi
@stealthanthrax
I see that you have changed the state machine architecture with the event queue model.
I have no more queries about the change and it looks good to me :rocket:
Norbert Preining
@norbusan
The event queue is only used for the planned actions, not for anything else. That way we don't need async calls anywhere.
all the rest is starting the whole bunch with start which listens for the keyword, and if it is found, jumps into the keyword callback. From there the STT and susi server connection is done, and answers processed. After this one returns to the start function which does a while True
Norbert Preining
@norbusan
Report on RPi4 with DeepSpeech 0.6.0: abysmal - extremely slow, detection is like, well, every third word might be correct ... so definitely a no go for now...
Nguyễn Hồng Quân
@hongquan
@norbusan Nice
Pranav Kulshrestha
@pranav1698

Hello all mentors and contributors, I wanted to contribute to this project since I have significant experience working on Linux projects (like meilix) before and wanted to get involved with this project. So I started going through the projects and found that they are many repositories related to susi_linux.
There are some questions that I wanted to ask moving forward,

What are some elementary steps working towards the project?
I have successfully installed susi_linux desktop client on my workstation using susi_installer and found some issues that were faced by me and some students participating in Google Code-In. I would like to work on these issues.
Are we following the same approach for raspi because I couldn't find a separate section for installation in raspi? (Or a separate disk image for raspi).

What are some issues that we are currently facing?
I would be very happy if I could help in solving some specific issues that we are currently facing in the project.

Report on RPi4 with DeepSpeech 0.6.0: abysmal - extremely slow, detection is like, well, every third word might be correct ... so definitely a no go for now...

I have worked with Deepspeech stt on raspi and we found that it is much better than other stt clients that we tested for our patent.

What are the plans for the project’s future?
@mariobehling told me that we are working on building SUSI.AI Linux distribution (something like meilix). What is the progress of that idea? I see many issues here https://github.com/fossasia/labs.fossasia.org/labels/SUSI%20AI that are open for susi_linux. Which issue are we currently working on? What issues are we targeting for this year’s summer?

Norbert Preining
@norbusan

@pranav1698 thanks for your interest. Concerning the issues, please open an issue at the susi_linux project for issues related to the actual client, and susi_installer for issues related to the installation and RPi setup.

If you want to see how we build RPi images, take a look at .travis/create-image script in susi_installer, it is more or less mounting the base image, installing prerequisites, and then running the susi installer with the necessary options.

Concerning your deepspeech experience, I am very interested to see more details. I have added deepspeech support to the python speech_recognition library, forked in the fossasia github org, and tried it on my RPi4, and unfortunately detection of words was really bad. I would love to have someone with more knowledge about deepspeech/rpi. On the desktop it works quite nice.

Concerning the future of SUSI.AI: I think at the moment there are two big open problems: The susi server has some issues, in particular on the RPi at the moment, which holds back new releases. The other is integration with desktop environments. And finally, good skills.

I am doing most of the susi_linux and susi_installer work, so feel free to ping me whenever you have a questions.

Pranav Kulshrestha
@pranav1698
@norbusan I will do the same,
Also, we used deepspeech on rpi3. I cannot share the script that I wrote for using deepspeech due to copyright issues, but it was working well as compared to other similar clients like pocketsphinx when we tested it for our use-case. I personally feel that if we could run Google/IBM stt modules locally then the output would be much better but there were time limits on our previous project so we were not able to move forward on that idea.
Norbert Preining
@norbusan
@pranav1698 it would be great if you could look into the recognize_deepspeech function I added to speech_recognition: https://github.com/fossasia/speech_recognition/blob/e452e9f3295232a6f5de0dc789acf2e1a4311f5c/speech_recognition/__init__.py#L846 and let me know whether there is anything that needs to be improved?
Concerning the Google/IBM stt modules, I don't think you can run them locally, you need to connect to the server (speech_recognition does that, and susi supports selecting google/watson/...)
Pranav Kulshrestha
@pranav1698

@pranav1698 it would be great if you could look into the recognize_deepspeech function I added to speech_recognition: https://github.com/fossasia/speech_recognition/blob/e452e9f3295232a6f5de0dc789acf2e1a4311f5c/speech_recognition/__init__.py#L846 and let me know whether there is anything that needs to be improved?

Sure

Norbert Preining
@norbusan
BTW, I am aware that on the RPi this line prot_buffer_file = os.path.join(language_directory, "output_graph.pbmm") is wrong, because the .tflite needs to be loaded.
@pranav1698 thanks!
Norbert Preining
@norbusan

Just for own memory, I finally understand why we got repeated hotword detections: The callback was only a rx next subject, thus immediately returned. This events were all queued by rx, and delivered after the initial callback (subscribe) was finished. Thus, occurrences of "susi" in the answer triggered new reactions via hotword detection.

Many of the problems appeared due to the async planned actions callbacks, but as well as via the wake button callbacks.

I think the proper solution is also not the _simple.py stuff I was doing, because it requires stopping and restarting the detection, which is suboptimal from efficiency POV. What we need is an actor model (pykka) where the hotword detection actor sends a message to the STT actor etc etc, and everything works async. That way we can deal with planned actions just as another message in the queue.

Maybe there are other solutions, I am thinking, but this is what I currently plan.

Norbert Preining
@norbusan

MOre details: the problem is with PyAUdio that is used in callback mode in a separate thread, so "susi" calls during processing are still recorded in snowboy_detector (into the ringbuffer), and thus processed after the current callback terminates.

This means, we either need to rewrite the snowboy_detector code to not use pyaudio in callback mode, or do something else ...

I am contemplating simply waiting for say 0.3s after returning from the callback and ignore all hotword detections in this time. Not optimal. Thinking ...

Pranav Kulshrestha
@pranav1698
fossasia/susi_installer#105
@norbusan can you review this one
Norbert Preining
@norbusan
@pranav1698 thanks for the fixed, reviewed and merged.

For the record, I have now fixed it by waiting 0.05s (that seems to be enough) after the return before putting the device into idle mode again. This way we can have the recognizer run permanently and still ignore susi words during normal answers.

If we ever change to a more complicated and 3-syllable hotword, this should be not necessary I guess.

Michael Christen
@Orbiter
sounds very interesting, can I get a new image to test the latest version?
Norbert Preining
@norbusan
@Orbiter on github ...
I am kicking of a new build with the configurable hotword stuff, and the susi server fix you added
Norbert Preining
@norbusan

@/all I have tagged a new release on github https://github.com/fossasia/susi_installer/releases/tag/build-20200108.1
This release works quite well for me on my RPi4. Besides having fixed server, fixed client, there is also a new feature: one can change the hotword my

~/SUSI.AI/bin/susi-config set hotword.model=computer.pmdl
sudo systemctl restart ss-susi-linux@pi

This would switch to the hotword "computer". Other models are Attention.pmdl, Robot.pmdl, jarvis.pmdl, and the default susi.pmdl

Please give it a try, both on the RPi and on the Desktop, but use the development branches on the desktop.

Michael Christen
@Orbiter
Oh wow that's good!
Michael Christen
@Orbiter

I just tested

~/SUSI.AI/bin/susi-config set hotword.model=computer.pmdl
sudo systemctl restart ss-susi-linux@pi

and its working great!

Norbert Preining
@norbusan
Good to hear, thanks for testing.
Mario Behling
@mariobehling

The event queue is only used for the planned actions, not for anything else. That way we don't need async calls anywhere.

@norbusan Right now it is only possible to set "Planned Actions". It is not possible to unschedule/delete them. It should be possible to "cancel all alarms", "cancel a planned action tomorrow" (but keep others) etc. - just like in a calendar. What are your ideas to achieve this?

Norbert Preining
@norbusan

@mariobehling First of all, planned actions are anyway broken at the moment, either because the server answer changed, or the implementation was broken from the beginning. I realized that while rewriting the engine. See this comment in the source code

                   # TODO TODO
                    # plan_delay is wrong, it is 0, we need to use
                    # plan = {'language': 'en', 'answer': 'ALARM', 'plan_delay': 0,
                    #         'plan_date': '2019-12-30T13:36:05.458Z'}
                    # plan_date !!!!!
                    self.action_schduler.add_event(int(plan['plan_delay']) / 1000, plan)

Then, concerning cancelling planned actions. I think this can be done, but we need:

  • a skill/server support so that the server answers properly
  • maybe types of planned actions to be deleted?

Then I can implement this easily by going through the Queue used for planned events and drop those that are to be cancelled.

Norbert Preining
@norbusan

@mariobehling it seems that my TODO there is actually wrong, and the code works as is, but the answers of the susi server are strange:

>>> import susi_python as susi
local server is down
>>> susi.ask("Set alarm for 5 minutes")
{'planned_actions': [{'language': 'en', 'answer': 'ALARM', 'plan_delay': 0, 'plan_date': '2020-01-09T01:59:53.199Z'}], 'language': 'en', 'answer': 'alarm set for for 5 minutes'}
>>> susi.ask("Set alarm in 5 minutes")
{'planned_actions': [{'language': 'en', 'answer': 'ALARM', 'plan_delay': 300001, 'plan_date': '2020-01-09T02:05:10.377Z'}], 'language': 'en', 'answer': 'alarm set for in 5 minutes'}
>>>

I get planned actions in both cases, but one with 0 delay that is immediately. Maybe this is only a question of skill development ...

Norbert Preining
@norbusan

@hongquan @stealthanthrax I have started to rework the layout of susi_linux into a more "standard" form:

  • the main module is not main but susi_linux
  • modules with only one file aren't put into sub-directories
  • removal of unused stuff
  • drop traces of state machine, rename susi_state_machine_simple.py to susi_loop
  • ...

The branch is here: https://github.com/norbusan/susi_linux/tree/layout-rework

What do you think about this?

Mario Behling
@mariobehling
@norbusan Yes, keep going :-)
Norbert Preining
@norbusan
@mariobehling Do you want to say I should straight away merge it?
Norbert Preining
@norbusan
One reason I want to rework is that I can provide a setup.py and the usual packaging infrastructure so that we can put the package on PyPy where people can easily install it with pip install SusiLinux or similar.
Mario Behling
@mariobehling
@norbusan Absolutely. That would be awesome! Of course you can merge it yourself. You are the core maintainer and core maintainers have the right to merge themselves. However, would be great to keep the issue and PRs mentioned in order to keep things transparent for others.
Pranav Kulshrestha
@pranav1698
@norbusan if you could open a issue regarding the same then I would like to work on that issue
Norbert Preining
@norbusan

@mariobehling I opened a PR fossasia/susi_linux#531 would be nice if someone can review it

@pranav1698 what do you mean concerning "work on that issue"? The branch is already prepared ;-) There can still be more things be done:

  • adding a setup.py with distutils integration
  • better dealing with config file config.json - I would like to move it to ~/.susi.config or similar instead of the CWD
  • after the previous step, rework the scripts and wrappers, so the wrappers become unnecessary
  • ...
Pranav Kulshrestha
@pranav1698
Thanks for the help @norbusan