by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 21 16:05
    chuwy labeled #152
  • Sep 21 16:05
    chuwy opened #152
  • Aug 27 15:39
    chuwy closed #151
  • Aug 27 15:39
    chuwy closed #149
  • Aug 27 15:39
    chuwy closed #150
  • Aug 27 15:39
    chuwy closed #147
  • Aug 27 15:39
    chuwy closed #146
  • Aug 27 15:39
    chuwy closed #145
  • Aug 27 15:39
    chuwy closed #132
  • Aug 27 15:39
    chuwy closed #148
  • Aug 27 15:39
    chuwy closed #143
  • Aug 27 15:39

    chuwy on master

    Bump jackson-databind to 2.10.3… Remove exclusiveMinimum and exc… Bump slf4j to 1.7.30 (close #14… and 6 more (compare)

  • Aug 27 12:26

    chuwy on gh-pages

    updated site (compare)

  • Aug 27 12:20

    chuwy on 1.0.2

    (compare)

  • Aug 27 12:00
    chuwy synchronize #148
  • Aug 27 12:00

    chuwy on 1.0.2

    (compare)

  • Aug 27 11:54

    chuwy on gh-pages

    updated site (compare)

  • Aug 27 11:46

    chuwy on 1.0.2-M5

    bump (compare)

  • Aug 27 11:45

    chuwy on gh-pages

    updated site (compare)

  • Aug 27 11:44
    chuwy synchronize #148
Fran Perez
@FPerezP
I tried first through the docker-compose

I actually tried:

  • docker compose
  • dockerfile
  • jar
  • build my own

All of them are failing connecting to Postgres, but I don't understand why. I'll keep trying and will keep you posted with what I'm doing wrong. Thanks for your help @chuwy!

Anton Parkhomenko
@chuwy
Can you connect to your postgres via psql?
Fran Perez
@FPerezP
yep, and I've checked that the sp_user exists and I granted it with SUPERUSER privs just in case
Anton Parkhomenko
@chuwy
Hm, this is very odd
Maybe there's a subtle typo in config file? Copy-pasting creds from config could help to identify this kind of mistakes
And they all like this one?
FATAL: role "X" does not exist org.postgresql.util.PSQLException: FATAL:
Fran Perez
@FPerezP
The only difference is that I had to modify the image from image: iglu-server:0.3.0 to image: snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.3.0
Anton Parkhomenko
@chuwy
That shouldn't have this kind of effect. I'm particularly baffled that this happens with plain jar
Fran Perez
@FPerezP
it's weird to me as well. No worries, I'll keep trying, don't want to bother too much :)
Anton Parkhomenko
@chuwy
That's fine! Keep us posted!
Fran Perez
@FPerezP
Thanks a lot! Will do! Really appreciate your interest and help!
Fran Perez
@FPerezP
Hey @chuwy I finally made it run through docker-compose, but I had to change a bit the PostgreSQL service definition:
    postgres:
        image: postgres
        hostname: postgres
        ports:
          - "5432:5432"
        environment:
          POSTGRES_USER: 'sp_user'
          POSTGRES_PASSWORD: 'sp_password'
          POSTGRES_DB: 'igludb'
and change the application.conf file to point postgres instead of localhost
it should be the same, but I don't know why it wasn't applying the init.sql script properly.
Anton Parkhomenko
@chuwy
Interesting
Hey @oguzhanunlu! Wdyt?
Oguzhan Unlu
@oguzhanunlu

Hi @FPerezP , I think in docker-compose case, iglu-server tried connecting to postgres before postgres's init.sql is run, hence the connection error. This possibility went unnoticed and will be fixed. See snowplow/snowplow-docker#60 .

Did your application.conf contain postgres.host as localhost during previous attempts? It should be postgres, since for a container, localhost isn't where you run docker-compose but the container itself.

Regarding your last successful attempt, race condition still stands and this time you got lucky. It isn't related to using environment variables or having init.sql.

Fran Perez
@FPerezP
thanks for the deeper insights @oguzhanunlu! I tried with both localhost and postgres (I've been doing tons of tests hehe). But trying with the default config and docker-compose.yml that are in Github was producing the error.
Anyhow, it's working fine now, I've stopped and started it quite a few times and it always worked :)
Oguzhan Unlu
@oguzhanunlu
Yes, as stated in the issue, current iglu server docker-compose example contains this race condition.
but, let me share a quick solution with you
Fran Perez
@FPerezP
thank you so much!
Oguzhan Unlu
@oguzhanunlu
Thanks for sharing with us!
Saeed Zareian
@szareiangm
@FPerezP I created my own docker with disabling the unit tests assuming that they are fine
this is my build command: sbt 'set test in assembly := {}' clean assembly
Saeed Zareian
@szareiangm
@chuwy @oguzhanunlu Hey, Iglu server cannot convert jsonshema to redshift schema. right? only iglu-ctl can do it?
Oguzhan Unlu
@oguzhanunlu
Hi @szareiangm , yes, iglu server only hosts your json schemas. Igluctl is used to do that conversion.
Jethro Nederhof
@jethron
Friggin' loving igluctl static deploy! Awesome work @oguzhanunlu!
Patrick Oberherr
@poberherr

:wave:

I am linting with igluctl 0.4.1 against a schema and I just get

1. Optional field doesn't allow null type

I know what this means and how to potentially debug this - my question is more :

Can we add the line number to debug that ? it's supper annoying to go through bigger schemas and do all the manual checking

Anton Parkhomenko
@chuwy
Hey @poberherr! Yes, it is available! Not exactly line number though, but JsonPointer and not as public release yet, but you can try out 0.6.0-rc1 which prints much more useful error messages
Patrick Oberherr
@poberherr
wow thanks a lot
Anton Parkhomenko
@chuwy
No problems. Let us know if there's any issues
Patrick Oberherr
@poberherr

So will test that;
On another note, I am looking more into programmatic testing of schemas, since kibana seems not to be feasible to make front-end developers test their schemas, so there is a bunch of stuff I need to tackle;

1) CI workflow to just check in the schemas
but for now I am interested in
2) People from the "frontend" chapter didn't want to have integration tests; so one idea which came up was: unit tests generated from the schema registry which you'd be able to test your schemas against

I was just wondering if something came up with you guys; or you have different approaches to that;
Personally I thought of something like snowplow/micro where you can call per event_id and see if the event was validated or not ( so Integration tests )
Anton Parkhomenko
@chuwy

So do you mean you want code that given:

  1. Schema (Iglu URI)
  2. Instance
  3. Registry

Will tell you wether instance is valid against the schema?

Patrick Oberherr
@poberherr
yes - to enable proper unit testing
( have to go to lunch now )
But I believe there are two sides to that;
  • Front end developers or "non tracking devs" who need to be able to write unit test for their custom schemas on the one hand
  • Snowplow "maintainers" to be able to drill down app_id / schema version to monitor where events are validated or not ( probably we can skip this point if the other point is solved )
Anton Parkhomenko
@chuwy
Okay. So the code you're looking for called "iglu clients". These are language-specific software though and right now we have just four of them: https://github.com/snowplow?utf8=%E2%9C%93&q=iglu+client&type=&language=
Probably you're looking for JS one, but I don't have any evidence somebody is using it
Patrick Oberherr
@poberherr
Will check this out after the break
Anton Parkhomenko
@chuwy
iglu-scala-client is cannonical and this is actually the one that invalidates all your events, so in your unit tests you would get exactly same error messages you're having in bad rows
Patrick Oberherr
@poberherr

hmm - okay - but then I have a follow up question ( and I am not sure if here at all )
So if the iglu-scala-client invalidated the events - this one would need to be rewritten to make the invalidated events more "usable" ?

For example to sort invalidated events by:

  • app_id
  • event_type
  • error message
Anton Parkhomenko
@chuwy
That's close, but not exactly. Iglu itself works only with self-describing entities (in our case contexts and unstructured events). It knows nothing about snowplow-specific entities such as app_id, event_id etc
The library that would have to be improved in order to provide that is Scala Common Enrich. And we're working on it
But Scala Common Enrich uses Iglu Client heavily, so its rather they both need to be improved
Patrick Oberherr
@poberherr
Ok gotcha, but this would be great news; I will take some notes and eventually I can help to contribute a bit - this is one of my major snowplow pain points
Francisco Albert Albusac
@tatitati
hi people, I have a question:
how different are self-describing json from json-schemas?....I'm trying to understand why iglu client is using self-describing json instead of json-schema (if the question make sense....). I'm reading from here: https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/, also I was reading that iglu-client only support self-describing json schemas