Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Viliam Durina
    @viliam-durina
    @peterchenadded Hmm, HashMap doesn't normally serialize HashMap$Node, that's an internal object. Can you share the full stack trace?
    peterchenadded
    @peterchenadded
    2020-08-14 17:02:25,220 [hz.dazzling_noyce.cached.thread-4] ERROR c.h.jet.impl.MasterJobContext - Execution of job '04cf-3091-ef80-0002', execution 04cf-2fb3-7ad4-0001 failed
    Start time: 2020-08-14T17:02:24.022
    Duration: 1,194 ms
    For further details enable JobConfig.storeMetricsAfterJobCompletion
    com.hazelcast.jet.JetException: Exception in SenderTasklet{ordinal=0, destinationAddress=[localhost]:5701, sourceVertexName='accumulate-properties'}: com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'java.util.HashMap$Node'
    at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.runTasklet(TaskletExecutionService.java:373)
    at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:891)
    at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.run(TaskletExecutionService.java:346)
    at java.lang.Thread.run(Thread.java:748)
    Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'java.util.HashMap$Node'
    at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:115)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:269)
    at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:378)
    at com.hazelcast.jet.impl.execution.SenderTasklet.tryFillOutputBuffer(SenderTasklet.java:152)
    at com.hazelcast.jet.impl.execution.SenderTasklet.call(SenderTasklet.java:112)
    at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.runTasklet(TaskletExecutionService.java:366)
    ... 3 common frames omitted
    Caused by: com.hazelcast.jet.JetException: Unable to serialize instance of class java.util.HashMap$Node: There is no suitable serializer for class java.util.HashMap$Node - Note: You can register a serializer using JobConfig.registerSerializer()
    at com.hazelcast.jet.impl.serialization.DelegatingSerializationService.serializationException(DelegatingSerializationService.java:138)
    at com.hazelcast.jet.impl.serialization.DelegatingSerializationService.serializerFor(DelegatingSerializationService.java:127)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:265)
    ... 7 common frames omitted
    Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: There is no suitable serializer for class java.util.HashMap$Node
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.serializerFor(AbstractSerializationService.java:507)
    at com.hazelcast.jet.impl.serialization.DelegatingSerializationService.serializerFor(DelegatingSerializationService.java:125)
    ... 8 common frames omitted
    HazelcastGitter
    @HazelcastGitter
    [Marko Topolnik, Hazelcast] if you have something like traverseIterable(hashMap.entrySet()) and try to emit that as the output in a flatmap satge, you'd get this error i think
    peterchenadded
    @peterchenadded
    @mtopolnik return Traversers.traverseIterable(distinctCorrelationIds.entrySet());
    @mtopolnik seems your spot on
    peterchenadded
    @peterchenadded
    @mtopolnik i guess there no default serializer for this and i would have to create my own? was working without issues in older version of jet (0.7)
    HazelcastGitter
    @HazelcastGitter
    [Marko Topolnik, Hazelcast] HashMap.entrySet() has an optimized, zero-allocation implementation, which means it exposes its internal map nodes to you. You must not let them escape the method.
    [Marko Topolnik, Hazelcast] the way to approach is it to map right away to another type, like Jet entry()
    [Marko Topolnik, Hazelcast] traverseIterable(map.entrySet()).map(e -> Util.entry(e.getKey(), e.getValue());
    peterchenadded
    @peterchenadded
    @mtopolnik ok will give that a try
    HazelcastGitter
    @HazelcastGitter
    [Marko Topolnik, Hazelcast] if you keep using and updating the correlationIDs after emitting it, then you should transform it entirely before passing on to the traverser
    [Marko Topolnik, Hazelcast] return traverseIterable(correlationIds .entrySet().stream() .map(e -> entry(e.getKey(), e.getValue())) .collect(Collectors.toList()));
    peterchenadded
    @peterchenadded
    @mtopolnik are there benefits to converts to Util.entry as there quite alot of places i would have to update. Is it simplier to add a HashMap$Node serializer?
    HazelcastGitter
    @HazelcastGitter
    [Marko Topolnik, Hazelcast] if you keep updating the map that you emit, you'll have to copy the contents anyway
    [Marko Topolnik, Hazelcast] you can always write a helper method that will be very simple to use in all places
    peterchenadded
    @peterchenadded
    @mtopolnik i checked it was too many places to manually update, have added a simple HashMap$Node serializer and errors gone now
    peterchenadded
    @peterchenadded
    @mtopolnik thanks for the awesome help, will do some performance testing to make sure no issues
    HazelcastGitter
    @HazelcastGitter
    [Marko Topolnik, Hazelcast] OK, good luck :)
    Can Gencer
    @cangencer
    Hi, we'll be decommissioning this channel soon - please join us at https://slack.hazelcast.com
    ArunKumarRajamandrap
    @charmingarun_twitter
    Hi Team,
    We implemented hazel cast jet as part of Java spring boot implementation, And that application will actively listens to 3 Kafka stream and update both maps and database(sql). This hazel cast jet cluster has 2 clients which actively listens to the maps.But whenever the cluster has throwing the exception one of the client is becoming down.
    Exception details :Ignoring heartbeat from Member [10.255.45.173]:5711 - 765b4aed-4011-45cf-9b37-be602025fbf9 since it is expired (now: 2020-12-21 20:41:19.035, timestamp: 2020-12-21 20:40:27.337)
    In order to remove the dependency what are the best options to resolve the issue. Will having back up or Split-Brain Protection will help the client getting the data ?
    please guide us
    Nicolas Frankel
    @nfrankel

    @charmingarun_twitter

    Hi, we'll be decommissioning this channel soon - please join us at https://slack.hazelcast.com