Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Apr 19 08:48
    benjben commented #153
  • Apr 17 11:00
    istreeter commented #153
  • Apr 16 21:03
    istreeter review_requested #153
  • Apr 16 21:03
    istreeter review_requested #153
  • Apr 16 20:48
    istreeter opened #153
  • Apr 16 20:41

    istreeter on blocker

    Ability to run blocking operatiā€¦ (compare)

  • Apr 16 20:40

    istreeter on develop

    (compare)

  • Apr 16 20:39

    istreeter on blocker

    blockerf (compare)

  • Sep 25 2020 14:07
    chuwy edited #152
  • Sep 21 2020 16:05
    chuwy labeled #152
  • Sep 21 2020 16:05
    chuwy opened #152
  • Aug 27 2020 15:39
    chuwy closed #151
  • Aug 27 2020 15:39
    chuwy closed #149
  • Aug 27 2020 15:39
    chuwy closed #150
  • Aug 27 2020 15:39
    chuwy closed #147
  • Aug 27 2020 15:39
    chuwy closed #146
  • Aug 27 2020 15:39
    chuwy closed #145
  • Aug 27 2020 15:39
    chuwy closed #132
  • Aug 27 2020 15:39
    chuwy closed #148
  • Aug 27 2020 15:39
    chuwy closed #143
Patrick Oberherr
@poberherr
I see - will go to the discourse forum - sorry!
Ben Fradet
@BenFradet
:+1:
Saeed Zareian
@szareiangm
Hey folks, a quick question.
Does Iglu server support SQL schemas to put in the registry? I see iglu-central has sql directory under its repo, but seems that Iglu server is only meant for JsonSchema. Did I understand it correctly?
Saeed Zareian
@szareiangm
validateSchema matches only format match { case "jsonschema" =>
and the rest are rejected.
Ben Fradet
@BenFradet
ping @chuwy @oguzhanunlu
Fran Perez
@FPerezP

Hi! Good morning everyone!
I'm trying to test Iglu locally. I tried first using the docker-compose the snowplow-docker repo provides, but I was having some Postgres issues, so I tried to build the jar file through sbt assembly, but I'm getting more Postgres errors related to role tester:
FATAL: role "tester" does not exist org.postgresql.util.PSQLException: FATAL: role "tester" does not exist

Have you faced this issue before? Is there anything I'm missing? Thanks a lot in advance for your help!

Anton Parkhomenko
@chuwy
Hey @szareiangm, sorry for delay. No, unfortunately server does not support it yet, but we definitely have plans to implement it. In fact at some point we just decided that it would be easier to go with static-repo+igluctl first, but initial plan was to generate all other DDLs/Schemas in server
Hi @FPerezP DB username (role) supposed to be in your configuration file, it is not hard-coded in Scala code for sure
Fran Perez
@FPerezP
Hey, thanks Anton, I've checked and it's in the test.conf file, but, do we need a Postgres instance up to run unit tests?
Anton Parkhomenko
@chuwy
Yes, I think few (integration though) tests indeed depend on running DB
Fran Perez
@FPerezP
:+1: thanks a lot! will check db users then :)
Anton Parkhomenko
@chuwy
But I'd go with provided docker image, @FPerezP. Or at least with built jar (https://bintray.com/snowplow/snowplow-generic/iglu-server#files). There's no need to build your own unless you want to implement something
Fran Perez
@FPerezP
I tried but I was getting some issues connecting to Postgres, so I was giving a try to the JAR way
I tried first through the docker-compose

I actually tried:

  • docker compose
  • dockerfile
  • jar
  • build my own

All of them are failing connecting to Postgres, but I don't understand why. I'll keep trying and will keep you posted with what I'm doing wrong. Thanks for your help @chuwy!

Anton Parkhomenko
@chuwy
Can you connect to your postgres via psql?
Fran Perez
@FPerezP
yep, and I've checked that the sp_user exists and I granted it with SUPERUSER privs just in case
Anton Parkhomenko
@chuwy
Hm, this is very odd
Maybe there's a subtle typo in config file? Copy-pasting creds from config could help to identify this kind of mistakes
And they all like this one?
FATAL: role "X" does not exist org.postgresql.util.PSQLException: FATAL:
Fran Perez
@FPerezP
The only difference is that I had to modify the image from image: iglu-server:0.3.0 to image: snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.3.0
Anton Parkhomenko
@chuwy
That shouldn't have this kind of effect. I'm particularly baffled that this happens with plain jar
Fran Perez
@FPerezP
it's weird to me as well. No worries, I'll keep trying, don't want to bother too much :)
Anton Parkhomenko
@chuwy
That's fine! Keep us posted!
Fran Perez
@FPerezP
Thanks a lot! Will do! Really appreciate your interest and help!
Fran Perez
@FPerezP
Hey @chuwy I finally made it run through docker-compose, but I had to change a bit the PostgreSQL service definition:
    postgres:
        image: postgres
        hostname: postgres
        ports:
          - "5432:5432"
        environment:
          POSTGRES_USER: 'sp_user'
          POSTGRES_PASSWORD: 'sp_password'
          POSTGRES_DB: 'igludb'
and change the application.conf file to point postgres instead of localhost
it should be the same, but I don't know why it wasn't applying the init.sql script properly.
Anton Parkhomenko
@chuwy
Interesting
Hey @oguzhanunlu! Wdyt?
Oguzhan Unlu
@oguzhanunlu

Hi @FPerezP , I think in docker-compose case, iglu-server tried connecting to postgres before postgres's init.sql is run, hence the connection error. This possibility went unnoticed and will be fixed. See snowplow/snowplow-docker#60 .

Did your application.conf contain postgres.host as localhost during previous attempts? It should be postgres, since for a container, localhost isn't where you run docker-compose but the container itself.

Regarding your last successful attempt, race condition still stands and this time you got lucky. It isn't related to using environment variables or having init.sql.

Fran Perez
@FPerezP
thanks for the deeper insights @oguzhanunlu! I tried with both localhost and postgres (I've been doing tons of tests hehe). But trying with the default config and docker-compose.yml that are in Github was producing the error.
Anyhow, it's working fine now, I've stopped and started it quite a few times and it always worked :)
Oguzhan Unlu
@oguzhanunlu
Yes, as stated in the issue, current iglu server docker-compose example contains this race condition.
but, let me share a quick solution with you
Fran Perez
@FPerezP
thank you so much!
Oguzhan Unlu
@oguzhanunlu
Thanks for sharing with us!
Saeed Zareian
@szareiangm
@FPerezP I created my own docker with disabling the unit tests assuming that they are fine
this is my build command: sbt 'set test in assembly := {}' clean assembly
Saeed Zareian
@szareiangm
@chuwy @oguzhanunlu Hey, Iglu server cannot convert jsonshema to redshift schema. right? only iglu-ctl can do it?
Oguzhan Unlu
@oguzhanunlu
Hi @szareiangm , yes, iglu server only hosts your json schemas. Igluctl is used to do that conversion.
Jethro Nederhof
@jethron
Friggin' loving igluctl static deploy! Awesome work @oguzhanunlu!
Patrick Oberherr
@poberherr

:wave:

I am linting with igluctl 0.4.1 against a schema and I just get

1. Optional field doesn't allow null type

I know what this means and how to potentially debug this - my question is more :

Can we add the line number to debug that ? it's supper annoying to go through bigger schemas and do all the manual checking

Anton Parkhomenko
@chuwy
Hey @poberherr! Yes, it is available! Not exactly line number though, but JsonPointer and not as public release yet, but you can try out 0.6.0-rc1 which prints much more useful error messages
Patrick Oberherr
@poberherr
wow thanks a lot
Anton Parkhomenko
@chuwy
No problems. Let us know if there's any issues
Patrick Oberherr
@poberherr

So will test that;
On another note, I am looking more into programmatic testing of schemas, since kibana seems not to be feasible to make front-end developers test their schemas, so there is a bunch of stuff I need to tackle;

1) CI workflow to just check in the schemas
but for now I am interested in
2) People from the "frontend" chapter didn't want to have integration tests; so one idea which came up was: unit tests generated from the schema registry which you'd be able to test your schemas against

I was just wondering if something came up with you guys; or you have different approaches to that;
Personally I thought of something like snowplow/micro where you can call per event_id and see if the event was validated or not ( so Integration tests )