Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 05 12:26
    spenes closed #234
  • Jun 29 22:42
    arthur78 opened #236
  • Jun 11 15:07

    istreeter on 2.0.6

    (compare)

  • Jun 11 15:06
    istreeter closed #235
  • Jun 11 15:06

    istreeter on master

    Bump jackson to 2.12.7 (close #… Prepare for 2.0.6 release (compare)

  • Jun 11 15:06

    istreeter on 1.0.7

    (compare)

  • Jun 11 15:06

    istreeter on 1.x

    Remove logging of user-sent rec… Bump amazon-kinesis-client to 1… Bump log4j-core to 2.17.1 (clos… and 6 more (compare)

  • Jun 11 15:06

    istreeter on 2.0.6

    (compare)

  • Jun 11 15:05

    istreeter on 2.0.6

    Prepare for 2.0.6 release (compare)

  • Jun 11 15:04

    istreeter on 2.0.6

    (compare)

  • Jun 11 15:04

    istreeter on 2.0.6

    Prepare for 2.0.6 release (compare)

  • Jun 11 15:03

    istreeter on 1.0.7

    (compare)

  • Jun 11 14:18

    istreeter on 1.0.7-rc1

    (compare)

  • Jun 11 14:18

    istreeter on 1.0.7

    Remove logging of user-sent rec… Bump amazon-kinesis-client to 1… Bump log4j-core to 2.17.1 (clos… and 6 more (compare)

  • Jun 11 14:16

    istreeter on 2.0.6-rc1

    (compare)

  • Jun 11 14:15

    istreeter on 2.0.6

    Bump jackson to 2.12.7 (close #… Prepare for 2.0.6 release (compare)

  • Jun 11 14:13
    istreeter opened #235
  • Jun 07 16:13
    arthur78 commented #234
  • Jun 07 13:55
    istreeter commented #234
  • Jun 07 11:19
    arthur78 edited #234
Ben Fradet
@BenFradet
Welcome to the Snowplow chat-room for the Snowplow ElasticSearch Loader! This is pure developer chat - for community support as always go to https://discourse.snowplowanalytics.com/ and for Snowplow Insights customer support, email support@snowplowanalytics.com
Saeed Zareian
@szareiangm
Hello @BenFradet !
Ben Fradet
@BenFradet
hey Saeed
Saeed Zareian
@szareiangm
Pleasure talking to you!
I want to ask two quick questions about your comments to address them.
Ben Fradet
@BenFradet
pleasure for me too, sure shoot :+1:
Saeed Zareian
@szareiangm
So now that you have created those issues, what should I do with them? because I have already committed.
Ben Fradet
@BenFradet
you just need to edit the commit messages
and squash the commits which belong to the same issue together
Saeed Zareian
@szareiangm
ah I see. git commit --amend
Ben Fradet
@BenFradet
yup you can also use rebase
Saeed Zareian
@szareiangm
sure thing. but do you need separate pull request for each issue?
Ben Fradet
@BenFradet
no
that's actually the point
just to have a single pull request solving multiple issues
Saeed Zareian
@szareiangm
super!
about this comment: same remark here -> should I get rid of RunTimeException and use either? Because I saw this pattern exists for other configuration errors.
also, queueConfig -> sourceSink. Our sink is elasticsearch and it is not the same as Enrich that its source and sink could be from similar type.
Ben Fradet
@BenFradet
actually you're right
on both counts
I think queue would be best
as repeating Config in queueConfig feels a bit weird to me
Saeed Zareian
@szareiangm
yeah. to me too. I tried to be coherent. but its cost for me is to look weird. lol.
excellent. I will put queue and will put a comment that we decided to use RunTimeException to keep the pattern.
works?
Ben Fradet
@BenFradet
yes use Either or Validation in place of exception
and at the very edge of your program you can throw only one exception
e.g. illegal argument exception
since it would have been misconfigured
Saeed Zareian
@szareiangm
okay. will do, squashing my commits now. I have another almost ready (fixing the unit-tests) pull request, right away after this one.
Ben Fradet
@BenFradet
cool :+1:
Saeed Zareian
@szareiangm
all comments are addressed
Ben Fradet
@BenFradet
hey Saeed, the commit history looks great, you just need to add (closes #{issue number}) at the end so that when we merge the issues will be closed automatically
Saeed Zareian
@szareiangm
done done :smile:
Ben Fradet
@BenFradet
thanks :+1: I'll have another full review tomorrow morning
Saeed Zareian
@szareiangm
excellent :+1:
Saeed Zareian
@szareiangm
@BenFradet Any remaining issues on my pull request?
Ben Fradet
@BenFradet
sorry I haven't had time to look at it yet, we've had issues with an upcoming release
it's on my list though :+1:
Saeed Zareian
@szareiangm
@BenFradet regarding the issue that I have opened (Abstracting the core):
the main package name is com.sp.elasticsearch.loader
how about changing it to com.sp.stream.loader? elasticsearch would a module anyways
Ben Fradet
@BenFradet
sounds good
Ben Fradet
@BenFradet
we might want to have a dedicated issue just for the package name change though
wdyt?
Saeed Zareian
@szareiangm
I will create the issue for it and then push both abstracting and and the package name change if my current PR gets approved.
ofaz
@ofaz

Hey all, I'm trying to get the latest docker container running, and it seems to have issues passing through the HTTP auth credentials to ES. I'm receiving the following:

[RecordProcessor-0000] ERROR com.snowplowanalytics.elasticsearch.loader.clients.ElasticsearchSenderHTTP - Shutting down application as unable to connect to Elasticsearch for over 500 ms
org.elasticsearch.client.ResponseException: POST https://$ESCLUSTER.aws.found.io:9243/_bulk: HTTP/1.1 401 Unauthorized
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/write/bulk] requires authentication","header":{"WWW-Authenticate":["Bearer realm=\"security\"","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"action [indices:data/write/bulk] requires authentication","header":{"WWW-Authenticate":["Bearer realm=\"security\"","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}
    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:354)
    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:343)
    at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
    at java.lang.Thread.run(Thread.java:748)

Am I doing something wrong?

In terms of the auth config in the config file, this is what mine looks like (with values swapped out)
 client {
    endpoint = "$CLUSTER.us-west-2.aws.found.io"
    port = "9243"
    username = "USER"
    password = "####"
    maxTimeout = "500"
    maxRetries = 20
    ssl = true
  }
ofaz
@ofaz
I rebuilt the JAR from source (rather than using the Docker images) and it seems to work now.
Ben Fradet
@BenFradet
mmh interesting, maybe there are issues signing requests in a container