Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 02 20:38
    scala-steward closed #2263
  • Dec 02 20:38
    scala-steward opened #2264
  • Dec 02 10:02
    scala-steward opened #2263
  • Dec 01 12:08
    Fraer edited #2262
  • Dec 01 12:05
    Fraer opened #2262
  • Dec 01 08:58
    sksamuel closed #2260
  • Dec 01 08:58
    sksamuel closed #2258
  • Dec 01 08:57
    sksamuel closed #2261
  • Dec 01 08:28
    scala-steward opened #2261
  • Nov 29 13:57
    scala-steward opened #2260
  • Nov 29 01:50
    sksamuel closed #2259
  • Nov 29 01:37
    scala-steward opened #2259
  • Nov 27 22:33
    scala-steward opened #2258
  • Nov 27 19:11
    stale[bot] closed #2215
  • Nov 27 08:45
    stale[bot] labeled #2220
  • Nov 27 08:45
    stale[bot] commented #2220
  • Nov 26 11:21
    Ekzer opened #2257
  • Nov 25 10:59
    izzyreal commented #2213
  • Nov 25 06:33
    sksamuel closed #2256
  • Nov 25 06:29
    scala-steward opened #2256
Christine Song
@cdisong_twitter
its either this error
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/ScalaObjectMapper org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:184) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:107) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.lambda$onTrigger$1(ReduceFnRunner.java:1057) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnContextFactory$OnTriggerContextImpl.output(ReduceFnContextFactory.java:438) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SystemReduceFn.onTrigger(SystemReduceFn.java:125) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.onTrigger(ReduceFnRunner.java:1060) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.emit(ReduceFnRunner.java:930) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.onTimers(ReduceFnRunner.java:790) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:95) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134) org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44) org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: Could not initialize class com.sksamuel.elastic4s.JacksonSupport$ org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:184) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:107) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.lambda$onTrigger$1(ReduceFnRunner.java:1057) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnContextFactory$OnTriggerContextImpl.output(ReduceFnContextFactory.java:438) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SystemReduceFn.onTrigger(SystemReduceFn.java:125) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.onTrigger(ReduceFnRunner.java:1060) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.emit(ReduceFnRunner.java:930) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.ReduceFnRunner.onTimers(ReduceFnRunner.java:790) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:95) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134) org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44) org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
i have both elastic-jackson and com.fasterxml.jackson.module imported
Domantas Petrauskas
@dope9967
@cdisong_twitter seems to me that you have incompatible jackson versions in the classpath, for example elastic4s pulls in 2.10 and some other dependency of your project pulls in 2.9 and you will get this error because of binary incompatibility. You got to figure out what dependencies are clashing and then upgrade/downgrade so jackson versions match.
also, this is unrelated to jackson support, because elastic4s core itself depends on jackson
Coderishi9
@Coderishi9
@sksamuel Sir how can I index a json file directly into elasticsearch using elastic4s
Daniel Joanes
@djoanes
Had to switch from Sttp to Java Async for basic auth -- will try to make a PR later if it becomes an issue
James Gorrie
@jamesgorrie

Hi all - I am really battling to see how to create and store search templates / stored scripts.

So far I have:
val myScript = script("scripts", search(indexName).query(MyQuery("{{query_param}}")).show).lang("mustache")

MyQuery is of type BoolQuery with the query of {{query_param}} set as we would put any other query.

That feels a little nasty as I am putting the query_param in via a string, and the search def needs an index, which you don't really need to construct search query JSON. These aside, once I have the script, I have no idea how to execute it?

When I try to run client.execute { myScript } I have no luck.

James Gorrie
@jamesgorrie
Looking through the codebase the only place we POST to _scripts is within the PutSearchTemplateRequest which is deprecated?
Lance Walton
@lancewalton
Hi all. I'm confused about what I'm supposed to use with elastic4s and AWS. I see that https://mvnrepository.com/artifact/com.sksamuel.elastic4s/elastic4s-aws_2.13/6.7.8 was released on the 22nd of July, but I've also seen other comments around saying that we can use elastic4s 7.8.0 and JavaClient.fromRestClient. But when I try the latter, I just get com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.net.ConnectException: Timeout connecting to [my aws elasticsearch URL].
Here's how I set up the ElasticClient:
ElasticClient(
          JavaClient.fromRestClient(
            RestClient.builder(urls.split(",").map(HttpHost.create(_)):_*).setRequestConfigCallback(
              new RestClientBuilder.RequestConfigCallback {
                @Override def customizeRequestConfig(requestConfigBuilder: RequestConfig.Builder): RequestConfig.Builder = {
                  requestConfigBuilder
                    .setConnectTimeout(60000)
                    .setConnectionRequestTimeout(60000)
                    .setSocketTimeout(60000)
                }
              }).build))
Vladislav Pakhomov
@d1skort
Hi all! What about merging zio update to 1.0? :) sksamuel/elastic4s#2187
1 reply
Lance Walton
@lancewalton
Regarding my issue above - it was nothing to do with elastic4s - apologies.
vonchav
@voonchav_gitlab
I'm hoping a new release of elastic4s will be cut soon with zio upgraded to 1.0.1. fingers crossed :)
1 reply
@ugobourdon
@ubourdon
Hi all. I upgrade my elastic4s from 1.7.x to 7.9.0 version (yep huge gap) and i wanted to know why embed Elasticsearch DB is not anymore contained in testkit (for TU) and what is the best/recommended alternative ?
2 replies
@ugobourdon
@ubourdon
No one can say me why embed module is removed form the project ?
Domantas Petrauskas
@dope9967
@ubourdon embedded servers are kind of meh when there are docker containers and libraries like https://github.com/testcontainers/testcontainers-scala, I'd suggest you just use that. Or start elasticsearch outside of tests, because it will take about 15-20s on each run.
bogdan sapizhak
@sapizhak

hi all!
i am getting some very strange behavior while using scripted sorting via elastic4s 7.6.0.

the script itself looks approximately like that:

if (doc['fieldName1'].size() != 0 && doc['fieldName2'].size() != 0) { // 1 - might not be true, so a fallback scenario is added
  long field1 = doc['fieldName1'].value;  
  double field2 = doc['fieldName2'].value;
  if (field1 == 2) {
    return field1 * params.paramValueName / field2 
  } else {
    return field2 * params.paramValueName / field1 
  }
} else {
  if (doc['fallBackField'].size() != 0) { // 2 - always true, checked for emptiness out of safety reasons
    return doc['fallbackField'].value; 
  } else {
    return 1; // least possible case
  }
};

each time a script is run, it ends in an arm 2 where the fallback scenario returns the value of a fallback field.
but when i query the index manually via curl by the known id, i can see that the values checked in the first if are always filled.

curl -X GET "localhost:9200/indexName_v46/_search?pretty" -H 'Content-Type: application/json' -d '
{ "query": { "bool": { "must" : [ { "match" : { "_id": "51c370c8-7a60-4b15-9466-0692e6b01e86" } } ] } }, "_source": [ "fieldName1", "fieldName2" ] }'

what might be the issue i receive null values on each call and fall into a fallback case?

Erik Wallin
@c01ac0ca_twitter
Hi, I am dependent on master (own pull request merged into master) for my upgrade to ES 7.9. Master version is 7.9.1-SNAPSHOT. I can only find up to 7.6 in snapshot repos and 7.9.1 is not released. Is i possible that someone at least triggers a snapshot build if you need more time until a release?
3 replies
@ugobourdon
@ubourdon
Hi. Do you know what use in Elastic4s DSL to clear a scroll ?
Diego Ferreira da Silva
@dfsilva

Hello Guys, this is how I am getting the results from agregations using elastic4s

val resultList = response.result.aggregations.data(name).asInstanceOfMap[String, Any].asInstanceOf[List[Map[String, Any]]]
val resultMap = resultList.map(obj => obj("key").asInstanceOf[A] ->
(obj("doc_count").asInstanceOf[B], obj("sum")
.asInstanceOfMap[String, C]
.asInstanceOf[C])).toMap[A, (B, C)]
.withDefaultValue((default1, default2))
values.map(id => (id, resultMap(id)._1, resultMap(id)._2))

Anyone knows a better way to do that?

Lance Walton
@lancewalton
Hi. I'm using version 7.9.1. I want to use the filter_path stuff shown here, https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-indexing.html, but I can't see how to do it with elastic4s. I'm doing a MultiSearchRequest if that makes any difference.
Lance Walton
@lancewalton
I think it might be sourceInclude... trying.
Lance Walton
@lancewalton
Seems to work.
kamisama-rney
@kamisama-rney
Hi, we're doing a file ingestion application that is currently trying to use the ES High Level client's BulkProcessor class. With its eager evaluation and asynchronous operation it's less than ideal for use within a FS2 stream for a Kafka --> S3 Read --> ES Insertion/Update stream. Since ES prefers bulk operations any suggestions accomplishing this without significant additions?
Julien BENOIT
@jbenoit2011
@sksamuel Would you be interested by a Http4s HttpClient implementation? We are currently testing one which we'll soon use in production, it seems fine AFAIK. I know http4s can be used through Sttp client but it is less straightforward, it does not work for Scala 2.13 as http4s-backend module for Sttp as not been published for Scala 2.13, requires to add sttp deps to your JAR. While with a direct Http4s integration, if the client is already used anywhere in the app, it can be reused for elastic4s as well.
2 replies
Andrew Valencik
@valencik
@jbenoit2011 I’d certainly be interested. We’re likely to be writing one in the next couple weeks, so having one upstream would be pretty lovely.
Johan Brodin
@brodin
@sksamuel any chance to fast-track this "bug" I found? 😅
sksamuel/elastic4s#2252
Michael Reynolds
@reynoldsm88
can the update api append an object to an array of of objects?
or rather...
i have a json structure like this
{
    document: "..."
    annotations: [
        {type: "one"},
        {type: "two"}
    ]
}
and i'd like to insert more of those nested "annotations" objects
passing in a Jackson annotated case class results in it trying to add ["three"]
Michael Reynolds
@reynoldsm88
converted it into a jackson object node results in it trying to append the value {type: "three"} as a string
and passing it in as a json string also converts it into a string
how can I update a document by appending a json object to an array of existing json objects?
Michael Reynolds
@reynoldsm88
would i be able to use indexable type classes?
Andrei Micu
@andrei-micu
Hey @reynoldsm88 ! Would this change help you? sksamuel/elastic4s#2244
Michael Reynolds
@reynoldsm88
@andrei-micu one sec let me check it out
thanks!!!
@andrei-micu yes that looks exactly like what i am looking for
all the behavior you describe there is what i ran into, namely that it either prefers to use primitives or it automatically unwraps case classes into arrays
and i couldn't understand why
Andrei Micu
@andrei-micu
@reynoldsm88 you're welcome! In the PR I have a message from @sksamuel that it will be reviewed soon, so I would hang tight till it reaches in a final form in the library. I don't see an easy way of going around the object parameters problem otherwise.
Michael Reynolds
@reynoldsm88
what i've done for the time being is not ideal but it works until there is a supported API solution
i just dropped down directly to the JSON API and used the underlying HttpClient
we're flexible though so once that feature is available we'll switch to that, sometimes you just gotta ship :/
thanks @andrei-micu
1 reply
Cosmin Simon
@cosmin-minna
hi Everyone! we updated our Elastic Search cluster to 6.8.13, where can I see what version of elastic4s works with that?
3 replies