Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Apr 19 08:48
    benjben commented #153
  • Apr 17 11:00
    istreeter commented #153
  • Apr 16 21:03
    istreeter review_requested #153
  • Apr 16 21:03
    istreeter review_requested #153
  • Apr 16 20:48
    istreeter opened #153
  • Apr 16 20:41

    istreeter on blocker

    Ability to run blocking operatiā€¦ (compare)

  • Apr 16 20:40

    istreeter on develop

    (compare)

  • Apr 16 20:39

    istreeter on blocker

    blockerf (compare)

  • Sep 25 2020 14:07
    chuwy edited #152
  • Sep 21 2020 16:05
    chuwy labeled #152
  • Sep 21 2020 16:05
    chuwy opened #152
  • Aug 27 2020 15:39
    chuwy closed #151
  • Aug 27 2020 15:39
    chuwy closed #149
  • Aug 27 2020 15:39
    chuwy closed #150
  • Aug 27 2020 15:39
    chuwy closed #147
  • Aug 27 2020 15:39
    chuwy closed #146
  • Aug 27 2020 15:39
    chuwy closed #145
  • Aug 27 2020 15:39
    chuwy closed #132
  • Aug 27 2020 15:39
    chuwy closed #148
  • Aug 27 2020 15:39
    chuwy closed #143
Anton Parkhomenko
@chuwy
Interesting
Hey @oguzhanunlu! Wdyt?
Oguzhan Unlu
@oguzhanunlu

Hi @FPerezP , I think in docker-compose case, iglu-server tried connecting to postgres before postgres's init.sql is run, hence the connection error. This possibility went unnoticed and will be fixed. See snowplow/snowplow-docker#60 .

Did your application.conf contain postgres.host as localhost during previous attempts? It should be postgres, since for a container, localhost isn't where you run docker-compose but the container itself.

Regarding your last successful attempt, race condition still stands and this time you got lucky. It isn't related to using environment variables or having init.sql.

Fran Perez
@FPerezP
thanks for the deeper insights @oguzhanunlu! I tried with both localhost and postgres (I've been doing tons of tests hehe). But trying with the default config and docker-compose.yml that are in Github was producing the error.
Anyhow, it's working fine now, I've stopped and started it quite a few times and it always worked :)
Oguzhan Unlu
@oguzhanunlu
Yes, as stated in the issue, current iglu server docker-compose example contains this race condition.
but, let me share a quick solution with you
Fran Perez
@FPerezP
thank you so much!
Oguzhan Unlu
@oguzhanunlu
Thanks for sharing with us!
Saeed Zareian
@szareiangm
@FPerezP I created my own docker with disabling the unit tests assuming that they are fine
this is my build command: sbt 'set test in assembly := {}' clean assembly
Saeed Zareian
@szareiangm
@chuwy @oguzhanunlu Hey, Iglu server cannot convert jsonshema to redshift schema. right? only iglu-ctl can do it?
Oguzhan Unlu
@oguzhanunlu
Hi @szareiangm , yes, iglu server only hosts your json schemas. Igluctl is used to do that conversion.
Jethro Nederhof
@jethron
Friggin' loving igluctl static deploy! Awesome work @oguzhanunlu!
Patrick Oberherr
@poberherr

:wave:

I am linting with igluctl 0.4.1 against a schema and I just get

1. Optional field doesn't allow null type

I know what this means and how to potentially debug this - my question is more :

Can we add the line number to debug that ? it's supper annoying to go through bigger schemas and do all the manual checking

Anton Parkhomenko
@chuwy
Hey @poberherr! Yes, it is available! Not exactly line number though, but JsonPointer and not as public release yet, but you can try out 0.6.0-rc1 which prints much more useful error messages
Patrick Oberherr
@poberherr
wow thanks a lot
Anton Parkhomenko
@chuwy
No problems. Let us know if there's any issues
Patrick Oberherr
@poberherr

So will test that;
On another note, I am looking more into programmatic testing of schemas, since kibana seems not to be feasible to make front-end developers test their schemas, so there is a bunch of stuff I need to tackle;

1) CI workflow to just check in the schemas
but for now I am interested in
2) People from the "frontend" chapter didn't want to have integration tests; so one idea which came up was: unit tests generated from the schema registry which you'd be able to test your schemas against

I was just wondering if something came up with you guys; or you have different approaches to that;
Personally I thought of something like snowplow/micro where you can call per event_id and see if the event was validated or not ( so Integration tests )
Anton Parkhomenko
@chuwy

So do you mean you want code that given:

  1. Schema (Iglu URI)
  2. Instance
  3. Registry

Will tell you wether instance is valid against the schema?

Patrick Oberherr
@poberherr
yes - to enable proper unit testing
( have to go to lunch now )
But I believe there are two sides to that;
  • Front end developers or "non tracking devs" who need to be able to write unit test for their custom schemas on the one hand
  • Snowplow "maintainers" to be able to drill down app_id / schema version to monitor where events are validated or not ( probably we can skip this point if the other point is solved )
Anton Parkhomenko
@chuwy
Okay. So the code you're looking for called "iglu clients". These are language-specific software though and right now we have just four of them: https://github.com/snowplow?utf8=%E2%9C%93&q=iglu+client&type=&language=
Probably you're looking for JS one, but I don't have any evidence somebody is using it
Patrick Oberherr
@poberherr
Will check this out after the break
Anton Parkhomenko
@chuwy
iglu-scala-client is cannonical and this is actually the one that invalidates all your events, so in your unit tests you would get exactly same error messages you're having in bad rows
Patrick Oberherr
@poberherr

hmm - okay - but then I have a follow up question ( and I am not sure if here at all )
So if the iglu-scala-client invalidated the events - this one would need to be rewritten to make the invalidated events more "usable" ?

For example to sort invalidated events by:

  • app_id
  • event_type
  • error message
Anton Parkhomenko
@chuwy
That's close, but not exactly. Iglu itself works only with self-describing entities (in our case contexts and unstructured events). It knows nothing about snowplow-specific entities such as app_id, event_id etc
The library that would have to be improved in order to provide that is Scala Common Enrich. And we're working on it
But Scala Common Enrich uses Iglu Client heavily, so its rather they both need to be improved
Patrick Oberherr
@poberherr
Ok gotcha, but this would be great news; I will take some notes and eventually I can help to contribute a bit - this is one of my major snowplow pain points
Francisco Albert Albusac
@tatitati
hi people, I have a question:
how different are self-describing json from json-schemas?....I'm trying to understand why iglu client is using self-describing json instead of json-schema (if the question make sense....). I'm reading from here: https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/, also I was reading that iglu-client only support self-describing json schemas
Oguzhan Unlu
@oguzhanunlu
hey @tatitati , sorry for the late response, I hope you found the reason in the meantime. If not, simply put, it is the self property at the top which describes the json schema. Let me also share that we have a discourse (https://discourse.snowplowanalytics.com/) which could be an option for future questions
Rajendra Kadam
@rkadam3_gitlab
Hello folks, I have some custom contexts defined in my iglu schema repository. What I would like to do is, I want to validate the context object when it is used in tracking in the code, and validate it against the schema defined. Any pointers to docs or way to do this would be super awesome and super helpful :pray: :bow: