Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Oct 04 14:16
    ysak-y opened #138
  • Aug 10 17:39
    dependabot[bot] labeled #137
  • Aug 10 17:39
    dependabot[bot] opened #137
  • Aug 10 17:39

    dependabot[bot] on npm_and_yarn

    Bump path-parse from 1.0.5 to 1… (compare)

  • Jun 10 16:37
    btburton42 commented #115
  • Jun 07 21:51
    jkelvie commented #115
  • May 09 23:41
    dependabot[bot] labeled #136
  • May 09 23:41
    dependabot[bot] opened #136
  • May 09 23:41

    dependabot[bot] on npm_and_yarn

    Bump hosted-git-info from 2.7.1… (compare)

  • May 07 01:17
    dependabot[bot] labeled #135
  • May 07 01:17
    dependabot[bot] opened #135
  • May 07 01:17

    dependabot[bot] on npm_and_yarn

    Bump underscore from 1.9.1 to 1… (compare)

  • May 06 20:48

    dependabot[bot] on npm_and_yarn

    (compare)

  • May 06 20:48
    dependabot[bot] closed #125
  • May 06 20:48
    dependabot[bot] commented #125
  • May 06 20:48
    dependabot[bot] labeled #134
  • May 06 20:48
    dependabot[bot] opened #134
  • May 06 20:48

    dependabot[bot] on npm_and_yarn

    Bump lodash from 4.17.15 to 4.1… (compare)

  • May 06 17:25
    dependabot[bot] labeled #133
  • May 06 17:25
    dependabot[bot] opened #133
ash-at-github
@ash-at-github
@jperata it runs fine locally (where we use 2.3.7 version of bespoken-tools) however only fails on server (which uses 2.1.22 version). Is that version no longer supported?
Juan Perata
@jperata
@ash-at-github I'm going to try out that version and see if i can reproduce the issue.
ash-at-github
@ash-at-github
ok, thanks @jperata
Meanwhile, how do we get request that virtual alexa sends? Is it via trace=true? You mentioned filter but could not find anything relevant to extract request using that https://read.bespoken.io/end-to-end/guide/#filtering-during-test Note that we use skillURL instead of handler as we use Java webservice
ash-at-github
@ash-at-github
@jperata Have some updates. Tried with 2.3.7 on server and it gave the same error there as well. So, you need not look into the version issue. When it runs on server, it fails with this error: "java.lang.SecurityException: Request with id amzn1.echo-external.request.ad2b1d6f-6158-47a6-bbfc-e361bb216815 and timestamp 1556915707000 failed timestamp validation with a delta of 35889
at com.amazon.ask.servlet.verifiers.SkillRequestTimestampVerifier.verify(SkillRequestTimestampVerifier.java:79) ~[ask-sdk-servlet-support-2.9.2.jar!/:?]" Basically timestamp validations are failing. Any idea about this?
ash-at-github
@ash-at-github
Even if we set to max timestamp validation value (150 seconds) per the documentation here: https://developer.amazon.com/docs/custom-skills/host-a-custom-skill-as-a-web-service.html#timestamp, looks like it will still fail? Assuming that the delta is 358 seconds per the error message above?
Juan Perata
@jperata
This is the request validation to verify the request is a proper request sent by Amazon. Our emulator can not generate a request that pass that validation, in order to continue your tests
you must either create an exception for things that comes from our test tools or test against a webhook that doesn't have that validation enabled
ash-at-github
@ash-at-github
We set the timestamp validation value to max and it worked. I just wonder why this issue would only happen on server when locally the tests run fine
Juan Perata
@jperata
are you sure that the validation is enabled on both?, doesn't make sense to have it enabled locally
ash-at-github
@ash-at-github
it's the same backend running on both places. Moreover this validation is done by Amazon SDK code and not part of our code
xcobbler
@xcobbler
Hello, I have a question about virtual-alexa. how can I do an Intent confirmation with this library on an intent that doesn't require confirmation (and hence doesn't have a dialog with the same name as the intent)? If I manually add a dialog to the model, the test framework does work, but through manual testing this breaks upsells.
Juan Perata
@jperata
Hi @xcobbler could you share how your intent look with and without the dialog in the model to be able to understand your issue a little better?
ash-at-github
@ash-at-github
We again started seeing "Invalid response: 400 Message: " for our virtual alexa scripts executed via Jenkins. We saw this before and we increased the timestamp validation limit to 150 seconds which is the max value suggested by Amazon (https://developer.amazon.com/docs/custom-skills/host-a-custom-skill-as-a-web-service.html#timestamp), which resolved that issue but it again started happening. Any suggestions?
Juan Perata
@jperata
Hi @ash-at-github , have you had any recent change in your jenkins environment or development server. A likely culprit, if it was working before, is that the timestamp has issues due the development server and jenkins being in different machines with their clocks on different times.
ash-at-github
@ash-at-github
no changes as far as we are aware, is there anyway to debug this on virtual alexa end?
Juan Perata
@jperata
if you are using virtual alexa directly as a Javascript library, you can use the filter property to log the requests and verify the timestamp.
If you are using virtual Alexa through YML tests, then you enable the "trace" property and it will print out the complete requests and responses for each interaction.
ash-at-github
@ash-at-github
@jperata Have used Trace option before, but how will it help in debugging this issue? to get the timestamps? We are getting timestamps right now after the error even without trace. And they seem ok. Anything else we need to use from trace?
Juan Perata
@jperata
yes, to get the timestamp that is being sent inside the request, and validate it against the server time in your jenkins server
Adam Elmore
@adamelmore
Hi Bespoken Team! Is it possible to use the virtual-alexa library to test interaction models only (for skills that don't use lambdas, and instead rely on external services)? I'm hoping to write unit tests to test the interaction model only; basically a bunch of "utter" tests where I'm asserting on intents and slots. Make sense?
John Kelvie
@jkelvie
Hi @adamelmore - it's likely possible with a bit of tweaking. But a quick question - what are you trying to test? Is it primarily making sure the interaction model is configured correctly? Or are you trying to make sure that the speech recognition and NLU are working right?
Adam Elmore
@adamelmore
I'll be looking into e2e later (with bespoken) to test speech reco, but right now I'm looking to build tests that assert that given an utterance, a specific intent is resolved with specific slot values. Maybe this isn't advised given that there may be a delta between what virtual-alexa resolves and what the actual Alexa would resolve?
John Kelvie
@jkelvie
Yes, there will definitely be a delta - our resolution mechanism is simplistic, and is only meant as a convenience mechanism
Adam Elmore
@adamelmore
One hack I'm considering is adding a fake handler that just parrots back the request as the response. In that way, I could assert on response.intent, etc.
John Kelvie
@jkelvie
It would be useful for ensuring all phrases you think are associated with an intent are actually associated with it - but that's about it. It won't tell you anything about ASR or NLU performance
Adam Elmore
@adamelmore
Cool, maybe I should just rely on e2e tests to validate that my interaction model is good and hasn't regressed.
John Kelvie
@jkelvie
The e2e is for full regression testing of code and AI - we actually have a new product that is squarely focused on testing just interaction models, in a very complete way
Adam Elmore
@adamelmore
oh, can you you point me to the interaction model product?
John Kelvie
@jkelvie
It's called Usability Performance Testing - our most recent case study on it is here: https://bespoken.io/blog/the-mars-agency-case-study/
Adam Elmore
@adamelmore
:tada: thanks a ton!
John Kelvie
@jkelvie
And a general overview is on our website: https://bespoken.io/usability-testing/
Our pleasure - and of course reach out if you have any additional questions
Adam Elmore
@adamelmore
Is there pricing information for usability-testing that I'm not seeing?
Or is it custom and you provide a quote?
John Kelvie
@jkelvie
It's the same pricing as for our subscription product: https://bespoken.io/pricing/ - that includes the usability performance testing
All of our products rely on interactions with Alexa/Google Assistant at their core, whether it is for end-to-end testing, monitoring or usability testing
xcobbler
@xcobbler
@jperata, amazon has acknowledged an issue with upsells when using certain models. my ugly workaround is to have 1 model for testing and 1 model for deployment :/
https://forums.developer.amazon.com/questions/207372/upsell-not-responding.html?childToView=210963#answer-210963
https://forums.developer.amazon.com/questions/208737/alexa-does-not-respond-to-request-for-upsell.html
Juan Perata
@jperata
thanks for keeping us informed @xcobbler , hopefully they will resolve the issue soon. But for now those links can help someone with the same issue in the future
Seth Lemoine
@mrseth01
I wrote a data library, and I'd like to write tests for it without emulating Alexa. I want to write tests that pull data from and endpoint and verify the data came back etc. How do I go about doing that?
I'm very new to testing. noob.
const NBA = require("../lambda/custom/nba-voice-library/NBA");

test("gets schedule from endpoint", async () => {
  expect.assertions(1);
  const data = await NBA.schedule();
  expect(data.length).toBe(4);
});
Juan Perata
@jperata
Hi @mrseth01 , your way sounds about right. Since you are not testing the Alexa section you don't need to introduce virtual-alexa in that section. We recommend that you also add some tests with our library to emulate Alexa too, it will interact with your endpoint unless you do some steps to mock it.
Mina Saleeb
@saleebm
Hey I'm getting an error about my network connection every time. I even tried following the node example but got this same error:
```
Something went wrong. Please check your network connection and try again.
Mina Saleeb
@saleebm
I'm using node v10, stack is RequestError: Error: read ECONNRESET\n at new RequestError ... t TLSSocket.socketErrorListener (_http_client.js:392:9)\n at TLSSocket.emit with macOS
John Kelvie
@jkelvie
Hi @saleebm - what tool are you using? And what command are you sending?
Mina Saleeb
@saleebm
Hey thank you for the quick response, I sent the command bst init and then I also tried node ./node_modules/bespoken-tools/bin/bst-proxy.js lambda ./src/index.ts
I installed bespoken-tools globally with npm v10.18.1 and also tried doing it in a local project,
Is it maybe because I use node version manager and I have my default set to v13?
Juan Perata
@jperata
Hi @saleebm - to use the proxy with the lambda parameters you would need to point to the ".js" file instead of ".ts" one.
Also, are you behind a corporate proxy?