Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Apr 19 08:48
    benjben commented #153
  • Apr 17 11:00
    istreeter commented #153
  • Apr 16 21:03
    istreeter review_requested #153
  • Apr 16 21:03
    istreeter review_requested #153
  • Apr 16 20:48
    istreeter opened #153
  • Apr 16 20:41

    istreeter on blocker

    Ability to run blocking operatiā€¦ (compare)

  • Apr 16 20:40

    istreeter on develop

    (compare)

  • Apr 16 20:39

    istreeter on blocker

    blockerf (compare)

  • Sep 25 2020 14:07
    chuwy edited #152
  • Sep 21 2020 16:05
    chuwy labeled #152
  • Sep 21 2020 16:05
    chuwy opened #152
  • Aug 27 2020 15:39
    chuwy closed #151
  • Aug 27 2020 15:39
    chuwy closed #149
  • Aug 27 2020 15:39
    chuwy closed #150
  • Aug 27 2020 15:39
    chuwy closed #147
  • Aug 27 2020 15:39
    chuwy closed #146
  • Aug 27 2020 15:39
    chuwy closed #145
  • Aug 27 2020 15:39
    chuwy closed #132
  • Aug 27 2020 15:39
    chuwy closed #148
  • Aug 27 2020 15:39
    chuwy closed #143
Alexander Dean
@alexanderdean
Welcome to the Snowplow chat-room for our Iglu open-source product! This is pure developer chat - for community support as always go to https://discourse.snowplowanalytics.com/ and for Snowplow Insights customer support, email support@snowplowanalytics.com
Patrick Oberherr
@poberherr
:wave:
Ben Fradet
@BenFradet
hey @poberherr
Patrick Oberherr
@poberherr
I see - will go to the discourse forum - sorry!
Ben Fradet
@BenFradet
:+1:
Saeed Zareian
@szareiangm
Hey folks, a quick question.
Does Iglu server support SQL schemas to put in the registry? I see iglu-central has sql directory under its repo, but seems that Iglu server is only meant for JsonSchema. Did I understand it correctly?
Saeed Zareian
@szareiangm
validateSchema matches only format match { case "jsonschema" =>
and the rest are rejected.
Ben Fradet
@BenFradet
ping @chuwy @oguzhanunlu
Fran Perez
@FPerezP

Hi! Good morning everyone!
I'm trying to test Iglu locally. I tried first using the docker-compose the snowplow-docker repo provides, but I was having some Postgres issues, so I tried to build the jar file through sbt assembly, but I'm getting more Postgres errors related to role tester:
FATAL: role "tester" does not exist org.postgresql.util.PSQLException: FATAL: role "tester" does not exist

Have you faced this issue before? Is there anything I'm missing? Thanks a lot in advance for your help!

Anton Parkhomenko
@chuwy
Hey @szareiangm, sorry for delay. No, unfortunately server does not support it yet, but we definitely have plans to implement it. In fact at some point we just decided that it would be easier to go with static-repo+igluctl first, but initial plan was to generate all other DDLs/Schemas in server
Hi @FPerezP DB username (role) supposed to be in your configuration file, it is not hard-coded in Scala code for sure
Fran Perez
@FPerezP
Hey, thanks Anton, I've checked and it's in the test.conf file, but, do we need a Postgres instance up to run unit tests?
Anton Parkhomenko
@chuwy
Yes, I think few (integration though) tests indeed depend on running DB
Fran Perez
@FPerezP
:+1: thanks a lot! will check db users then :)
Anton Parkhomenko
@chuwy
But I'd go with provided docker image, @FPerezP. Or at least with built jar (https://bintray.com/snowplow/snowplow-generic/iglu-server#files). There's no need to build your own unless you want to implement something
Fran Perez
@FPerezP
I tried but I was getting some issues connecting to Postgres, so I was giving a try to the JAR way
I tried first through the docker-compose

I actually tried:

  • docker compose
  • dockerfile
  • jar
  • build my own

All of them are failing connecting to Postgres, but I don't understand why. I'll keep trying and will keep you posted with what I'm doing wrong. Thanks for your help @chuwy!

Anton Parkhomenko
@chuwy
Can you connect to your postgres via psql?
Fran Perez
@FPerezP
yep, and I've checked that the sp_user exists and I granted it with SUPERUSER privs just in case
Anton Parkhomenko
@chuwy
Hm, this is very odd
Maybe there's a subtle typo in config file? Copy-pasting creds from config could help to identify this kind of mistakes
And they all like this one?
FATAL: role "X" does not exist org.postgresql.util.PSQLException: FATAL:
Fran Perez
@FPerezP
The only difference is that I had to modify the image from image: iglu-server:0.3.0 to image: snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.3.0
Anton Parkhomenko
@chuwy
That shouldn't have this kind of effect. I'm particularly baffled that this happens with plain jar
Fran Perez
@FPerezP
it's weird to me as well. No worries, I'll keep trying, don't want to bother too much :)
Anton Parkhomenko
@chuwy
That's fine! Keep us posted!
Fran Perez
@FPerezP
Thanks a lot! Will do! Really appreciate your interest and help!
Fran Perez
@FPerezP
Hey @chuwy I finally made it run through docker-compose, but I had to change a bit the PostgreSQL service definition:
    postgres:
        image: postgres
        hostname: postgres
        ports:
          - "5432:5432"
        environment:
          POSTGRES_USER: 'sp_user'
          POSTGRES_PASSWORD: 'sp_password'
          POSTGRES_DB: 'igludb'
and change the application.conf file to point postgres instead of localhost
it should be the same, but I don't know why it wasn't applying the init.sql script properly.
Anton Parkhomenko
@chuwy
Interesting
Hey @oguzhanunlu! Wdyt?
Oguzhan Unlu
@oguzhanunlu

Hi @FPerezP , I think in docker-compose case, iglu-server tried connecting to postgres before postgres's init.sql is run, hence the connection error. This possibility went unnoticed and will be fixed. See snowplow/snowplow-docker#60 .

Did your application.conf contain postgres.host as localhost during previous attempts? It should be postgres, since for a container, localhost isn't where you run docker-compose but the container itself.

Regarding your last successful attempt, race condition still stands and this time you got lucky. It isn't related to using environment variables or having init.sql.

Fran Perez
@FPerezP
thanks for the deeper insights @oguzhanunlu! I tried with both localhost and postgres (I've been doing tons of tests hehe). But trying with the default config and docker-compose.yml that are in Github was producing the error.
Anyhow, it's working fine now, I've stopped and started it quite a few times and it always worked :)
Oguzhan Unlu
@oguzhanunlu
Yes, as stated in the issue, current iglu server docker-compose example contains this race condition.
but, let me share a quick solution with you
Fran Perez
@FPerezP
thank you so much!
Oguzhan Unlu
@oguzhanunlu
Thanks for sharing with us!
Saeed Zareian
@szareiangm
@FPerezP I created my own docker with disabling the unit tests assuming that they are fine
this is my build command: sbt 'set test in assembly := {}' clean assembly
Saeed Zareian
@szareiangm
@chuwy @oguzhanunlu Hey, Iglu server cannot convert jsonshema to redshift schema. right? only iglu-ctl can do it?
Oguzhan Unlu
@oguzhanunlu
Hi @szareiangm , yes, iglu server only hosts your json schemas. Igluctl is used to do that conversion.