It's really easy to get someone defensive when people start asking for changes in what they have been working on for a while now. So in this case it's not entirely on dud here. Not to bring this down that path but just wanted to bring that up.
I hadn't targeted that at anyone in particular. There has just been a general vibe for a while lately that I wanted to get ahead of.
I'm aware that I can be a little blunt in my reviews, so it's a good reminder for myself too.
I just want to say that I appreciate all you guys, the time you take and spend on this, your feedback, and your opinions. So far I feel that all of it comes from a good place, and that I also appreciate Jacob Tomlinson's efforts to make sure it stays that way. Thanks to all of you.
Woohoo! I just merged the teams PR! Thank you Jacob Tomlinson and FabioRosado for figuring out how to add teams support!
Thanks for all the help reviewing. And your work on the test machinery. Tests have been the main blocker for the PR (and time) but that was a joy to test.
Maybe I'm missing something, but if I change this line https://github.com/opsdroid/opsdroid/blob/master/opsdroid/connector/github/tests/test_connector_github.py#L237 to
assert event.abc == "pyup-bot" and then run pytest opsdroid/connector/github/tests/test_connector_github.py::test_pr_merged it still passes. Any reason why this would be?
Oooh interesting. I think because the assertions are inside the skill they get ignored.
We log exceptions in skills and move on. Don't want user code to cause the whole thing to die. https://github.com/opsdroid/opsdroid/blob/a45490d1bdceca39b49880e20262b55ea0be101d/opsdroid/core.py#L466
So we should probably be asserting that
Exception when running skill is not logged once the skill has been run.
Good catch thanks!
Yes, that totally makes sense. Hmmmm, I understand this, but in terms of implementation, that's like saying we should put the
call_endpoint in a try and then catch the exception coming back, yah? If not, lemme know if you can pseudocode it out for me, and I'll try making the change
We need to add
caplog to the arguments here https://github.com/opsdroid/opsdroid/blob/a45490d1bdceca39b49880e20262b55ea0be101d/opsdroid/connector/github/tests/test_connector_github.py#L227
Then add
assert "Exception when running skill" not in caplog.text here https://github.com/opsdroid/opsdroid/blob/a45490d1bdceca39b49880e20262b55ea0be101d/opsdroid/connector/github/tests/test_connector_github.py#L249
Oho. Okay, I'll see if I can get that working for you. Thanks for much for the pointer.... learning :)
We probably need to do this in every test that has assertions inside the skill
Jacob Tomlinson: so I am not convinced the skill-based tests are working. Here is what I mean by this... I couldn't get the
caplog stuff working. But I read all about caplog and there's no reason it shouldn't work. So I tried just straight raising an exception in the skill, and finally just tried putting a breakpoint (like a pdb.set_trace()) in the skill... and none of it registered. There didn't seem to be any change I could make in the skill that would affect anything at all. I tried this with multiple skills. I only tried this with skills in the github connector so far.
Can you (or anyone for that matter) give this a try and see if you are getting something different from me? Maybe I'm doing something wrong.
I think you're right. I'm not sure the skills themselves ever get executed. Opsdroid adds the skill to the event loop and returns a 201 to let you know the execution has been created. But it doesn't wait for the execution to finish, this is because users could put all sorts in their skill and if it takes too long the GitHub webhook will time out.
Really we need something after the
assert resp.status == 201 which waits for the skill to be called. Maybe even adding an asyncio.sleep after will be enough to get it to hit your breakpoint. But we probably need some testing utility which blocks until the skill has been called.
Looks like there was a bug in the GitHub connector which did mean the skill was never called. I've opened opsdroid/opsdroid#1808 to fix it. I also updated the test to check that the skill was called.
With that bug fixed things are as we thought. The assertions in the skill fail but opsdroid continues anyway and the test passes. Checking the log and asserting that no exception was raised fixes that, so if you break the assertion the test fails. https://github.com/opsdroid/opsdroid/pull/1808/commits/d8dd73b9a70ba6198cc460740f17373f8f39a3aa
Yesssss, awesome. Good stuff! Also, glad it wasn’t something deeper as I had originally suspected. Will officially review shortly :)
Jacob Tomlinson: thanks to the path you paved, I added
caplog capability to all the other Github skills based tests: opsdroid/opsdroid#1811
Awesome thanks so much! Merging the first has made conflicts in the second. Once you've addressed those I'll have a look.
1 reply
Jacob Tomlinson: I think since you merged opsdroid/opsdroid#1820 we can just close opsdroid/opsdroid#1794
I have a question about the Github connector. I'm trying to make some modifications to it just to test some stuff out. I make my modifications and run opsdroid with a Github skill set up for the PRClosed event.
I close a PR in my repo which runs the skill. I can tell the skill is initially triggered because I see this message in my opsdroid log: 2021-08-22 23:41:33,955 INFO aiohttp.access: 127.0.0.1 [23/Aug/2021:04:41:33 +0000] "POST /connector/github HTTP/1.1" 201 191 "-" "GitHub-Hookshot/59a1e9d". That implies to me that this line was executed: https://github.com/opsdroid/opsdroid/blob/master/opsdroid/connector/github/__init__.py#L346
However, the code in my skill itself is never run. My guess is that due to the changes I made, there is an error somewhere up the chain in opsdroid itself before my actual skill is run. Something is failing, but there appears to be some sort of rescue in place so that the entire opsdroid itself doesn't die. If that's the case, there is no message being logged, so I can't tell where control goes after the line I pasted above.
Can anyone tell me if they think I'm even on the right path?
3 replies
Check the log for details.
Jacob Tomlinson: Are you kidding me? Yes, that was the problem. There was an error in what I was importing in the skill. FIxed that and now back to good. Thank you so much.
Do you see any opportunities to add some more logging anywhere to help others avoid this in the future?
1 reply
Hehe I’m looking into that exact thing :) Going to finish my current train of thought, and then recreate the issue and either get some logging going or suggest what might have helped.
Thanks again.
Maybe there's an opportunity to add something in the docs about this. For me, I think my expectation was that app would have just errored out and stopped working rather than just throw an error but continue on. I didn't realize that might be the case until i was literally typing my original message in this channel. I'll take a look and potentially make a suggestion if I think it would help.
Opsdroid doesn't crash out when a skill causes a crash when other skills are running. And sometimes not even then.
Yes, I get that now. :) And it totally makes sense. I'm just trying to think of what may help other developers who don't realize this when they are working with opsdroid in the future.
Yeah, I hear you. Opsdroid is very hard to debug for sure.
Well now that I get it, I'm having a great time 😀 But I wasn't familiar with opsdroid (or python really) before, so I was just easily confused
Yeah, I think everyone has that ah-ha moment. I definitely did.
I'd love to hear thoughts about how we can improve this, I totally get that it can be a pain. If your skill errors it shows in the log. I'm pretty liberal with breakpoints when I'm debugging. But what would folks like to see?
No it’s exactly what I would like to see. I just missed it. That’s totally my fault. That’s why I’m advocating for something in the docs for writing/understanding skills that implies that if your skill fails, you will get an error in the logs, but the app won’t necessarily die.