Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Jaromir Hamala
    @jerrinot
    so you could demote a full member to a light member before stopping an app
    lprimak
    @lprimak
    I tried that, but it brings in a lot of other challenges, for example, managing that it's the "only" lite member then the cluster data loss would occur.
    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    hi @ssahadevan , thanks a lot for information
    but in our application , our microservice is only sitting in kubernetes , hazelcast is sitting outside of it , so only we want to use TCP-IP mechanism
    Jaromir Hamala
    @jerrinot
    @Vignesh-Thiraviam can you share logs from all your Hazelcast cluster members?
    just to avoid any confusion: members: "10.173.14.23:5701,10.173.14.24:5701,10.173.14.25:5701,10.173.14.26:5701" says where to look for other members.
    It does not tell on what port this member should bind to. by default Hazelcast binds to 5701. If that port is unavailable it'll increment the port by 1 and will try again.
    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    ok thanks @jerrinot , even if we specify port number , it will still look for others ?
    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    just to confirm even if we specify 10.173.14.23:5701 , it will still look for other ports ? @jerrinot , because we are observing the same
    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    Also is there a way to specify the port @jerrinot , kindly help on this :)
    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    port:
      auto-increment: true
    is this the one you are telling @jerrinot , should we turn off this ?
    Jaromir Hamala
    @jerrinot
    if you change this to false then it will try to bind to the specified port only. when this port is unavaialbe (=already occupied) then Hazelcast will fail to start
    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    ok thanks @jerrinot , by default this is true ?
    Jaromir Hamala
    @jerrinot
    yes, it is true by default
    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    network:
    port:
    auto-increment: false
    join:
    multicast:
    enabled: false
    tcp-ip:
    enabled: true
    members: "10.173.14.23:5701,10.173.14.24:5701,10.173.14.25:5701,10.173.14.26:5701"
    is this now ok @jerrinot , will this work properly ?
    Jaromir Hamala
    @jerrinot
    so with this configuration it will:
    1. try to bind to a port 5701. when this port is unavailable then Hazelcast is will to start.
    2. if the port is available then the member starts. it will start looking for other cluster members and specified addresses. there are 2 options:
      2a. It will find an already running cluster on one of the configured addresses. then it will try to join the cluster. if the join is successful then it'll receive a list of other members running in the already existing cluster. important: the list it will receive from the existing cluster is not necessary the same as the addresses you configured.
      2b. it wont find any cluster running on either of the addresses. in this case the member will form its own cluster and it will start periodically checking the "10.173.14.23:5701,10.173.14.24:5701,10.173.14.25:5701,10.173.14.26:5701" and if eventually finds a cluster running there then it will try to join it. (if some other conditions are fulfilled)
    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    thanks a lot @jerrinot , so I will need to make sure that , all other members specified , doesn't have a different cluster to join . I will test this fix . Thank you
    Jaromir Hamala
    @jerrinot

    exactly. Othewise you can get surprising results.

    Consider this: You have 3 members: Member0, Member1, Member2

    Member0 has this configuration:

    hazelcast:
      network:
        join:
          multicast:
            enabled: false
          tcp-ip:
            enabled: true
            members: "Member0:5701, Member1:5701"

    Member1 has this configuration:

    hazelcast:
      network:
        join:
          multicast:
            enabled: false
          tcp-ip:
            enabled: true
            members: "Member0:5701, Member1:5701"

    Member2 has this configuration:

    hazelcast:
      network:
        join:
          multicast:
            enabled: false
          tcp-ip:
            enabled: true
            members: "Member0:5701"

    You start only Member0 and Member1. They have each other addresses in configuration so they discover each other and will form a cluster.

    Then you start Member2. It has only Member0 in its configuration, initially it knows nothing about Member1. So Member2 will discover an existing cluster running on Member0:5701 and will join this cluster. On joining it receives a list of all current cluster member. This list will contain also Member1:5701 so eventually Member2 will connect to Member1:5701 even it does not have its address in a configuration at all.

    I hope it makes some sense to you:)

    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    thanks a lot , so I willtry this configuration port:
    auto-increment: false
    thank you @jerrinot
    Jaromir Hamala
    @jerrinot

    broadcast

    Advertisement Warning
    Hazelcast is running a public Slack at https://slack.hazelcast.com/
    It allows a more structured discussions, with channel, threads, etc.

    Everyone is welcome to join!

    jayar
    @jayar
    Hi I have looked through the documentation about sizing when Hazelcast is deployed on Kubernetes, but I can’t find any information, only normal clusters. I have also tried to look into the helm charts but I couldn’t see any resource limits, so do you have some recommendations?
    Hazelcast
    @hazelcast_twitter
    [Sharath Sahadevan (Sharath Sahadevan)] @jayar: Please review the deployment and operations guide that you can download here - https://hazelcast.com/resources/hazelcast-deployment-operations-guide/ . There is a chapter on considerations for Hazelcast Cluster sizing . It is not kubernetes specific, but gives general guidelines , based on how much data you plan to store in Hazelcast and which features you plan to leverage for your use case.
    Shivam Sharma
    @svmsharma20

    Hi All,

    We are facing a considerable delay around 3-8 mins in receiving of messages published on hazelcast topic. Messages get broadcasted to all the nodes. At the subscribers, end node check using some field present in the message to decide whether it has to process it or not. If the node qualifies then it will process the message. Otherwise, it will ignore that message. In our case, every time the same node which publishes the message gets qualified and, that is expected as per our use case. But the problem is, the node which publishes the message itself is getting the message with the delay of 3-8 mins and, all other nodes are getting the same message on time that is in a fraction of seconds.

    We already checked the memory utilization and GC graph, they are normal, and also the hazelcast cluster is stable. Also, this issue occurs sporadically 4-5 times a day so, it's not reproducible in house.

    Any idea why Hz is behaving like this? Is anything we can do to diagnosis the issue? Is there any workaround which we can do? Any suggestions would be very helpful.

    Thanks

    Hazelcast
    @hazelcast_twitter
    [Peter Veentjer (peter.veentjer)] This is very weird behavior because messages get send immediately to all nodes
    [Peter Veentjer (peter.veentjer)] one of the things that could make it possible if the event thread responsible for processing that message is busy e.g. blocked
    [Peter Veentjer (peter.veentjer)] since the the event thread runs alien code, the blocking could be caused by alien code
    [Peter Veentjer (peter.veentjer)] so the first thing I would do is to track the time the event handlers are running.. e.g. when it takes more than a second, you add a system.out
    [Peter Veentjer (peter.veentjer)] also it would help to run with a profiler, e.g. JFR and check what the threads are actually doing
    [Peter Veentjer (peter.veentjer)] the event threads can be seen from JFR
    [Peter Veentjer (peter.veentjer)] another potential cause could be delay in sending the message once it is push on the outbound pipeline due to networking problems. However 3-9m delay; is too much. The connection would be closed by then
    Hazelcast
    @hazelcast_twitter
    [Peter Veentjer (peter.veentjer)] FYI: The event threads are shared between all topics (and a few other mechanisms like collection events). So maybe you have some slow code the handlers of other topics or some slow collection handler which causes another topic to become slow.
    ihorpts
    @ihorpts

    hi all
    I have one question about JCache declarative configuration in hazelcast. I've read documentation but somehow I can't find an answer yet, so I'd much appreciate any help.

    I'd like to configure my caches kind of dynamically (it applies mostly to hazelcast client but for server side probably should be the same). I need to join existing cluster (given hazelcast client/server instance with it own configuration) and then get cache manager with a declarative configuration of needed caches, something like this:

    //server
    HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(hzConfig);
    CachingProvider caching = Caching.getCachingProvider("com.hazelcast.cache.impl.HazelcastServerCachingProvider");
    Properties properties = new Properties();
    properties.setProperty( HazelcastCachingProvider.HAZELCAST_INSTANCE_NAME, hazelcastInstance.getName());
    CacheManager cacheManager = caching.getCacheManager(new URI("classpath:hz-my-cache-test.xml"), null, properties);
    Cache<Long, Object> cache = cacheManager.getCache("my-cache-name");

    The same direction is for the client but with proper client caching provider, etc.

    Is it possible at all?

    Somehow from [THIS] (https://docs.hazelcast.org/docs/latest/manual/html-single/#scoping-to-join-clusters) hazelcast documentation I didn't get it and my prototype app doesn't read such configured caches. Although putting cache configuration directly into hazelcast.xml works just fine.

    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    Hi All , is there any provision to secure Hazel cast man center , clustered REST
    Emre Aydın
    @emre-aydin
    Hi @Vignesh-Thiraviam . We don't have any such plans at the moment. I will contact our product manager to add it to the roadmap. Can't make any promises about the timeline though.
    Emre Aydın
    @emre-aydin
    BTW, I just saw your earlier message regarding disabling some REST endpoints in Management Center. It's not possible to do so at the moment.
    Inel Pandzic
    @inelpandzic
    Hey everyone, I have one question. I can't see what is the cause for: Still have pending migration tasks, cannot lock cluster state! New state: ClusterStateChange{type=class com.hazelcast.cluster.ClusterState, newState=FROZEN}, current state: ACTIVE?
    Vignesh-Thiraviam
    @Vignesh-Thiraviam
    Hi @emre-aydin , thanks a lot for the information. Will plan to secure it externally
    Thanks a lot , regarding for REST endpoints too
    jklingsporn
    @jklingsporn
    Hello, I am trying to create a hazelcast cluster using AWS ElasticBeanstalk and even though I've (IMHO) configured everything correct, the servers cannot connect to each other (even though I've setup the security groups properly (as described in Hazelcast_IMDG_Amazon_EC2_Deployment_Guide_SPOT_Letter_v2.5)). The nodes find each other, so the IAM-Role was granted with the proper rights. Unfortunately I cannot use the latest version, as I am bound to Hazelcast 3.12 so I am refering to https://github.com/hazelcast/hazelcast-aws/tree/2.4.x.
    I've seen this open issue though: hazelcast/hazelcast-aws#69
    Hazelcast
    @hazelcast_twitter
    [Rafal Leszko (unknown)] @jklingsporn can you try asking in the same GH Issue?
    jklingsporn
    @jklingsporn
    I already made a comment yesterday
    it is very strange to me as ElasticBeanstalk is using EC2 under the hood
    Is it probably because I am using t3.small instances and they only have up to 5GB/s network? Should I try increasing the connect timeout(Currently 10 secs)? But there is basically no data in the cluster and there are only 2 nodes.
    jklingsporn
    @jklingsporn
    Also: does it make a difference if I configure the cluster using the <aws>-tag or using a <discovery-strategy>?
    Mesut Celik
    @mesutcelik
    no difference but <aws> much cleaner and easier to use.