We are facing a considerable delay around 3-8 mins in receiving of messages published on hazelcast topic. Messages get broadcasted to all the nodes. At the subscribers, end node check using some field present in the message to decide whether it has to process it or not. If the node qualifies then it will process the message. Otherwise, it will ignore that message. In our case, every time the same node which publishes the message gets qualified and, that is expected as per our use case. But the problem is, the node which publishes the message itself is getting the message with the delay of 3-8 mins and, all other nodes are getting the same message on time that is in a fraction of seconds.
We already checked the memory utilization and GC graph, they are normal, and also the hazelcast cluster is stable. Also, this issue occurs sporadically 4-5 times a day so, it's not reproducible in house.
Any idea why Hz is behaving like this? Is anything we can do to diagnosis the issue? Is there any workaround which we can do? Any suggestions would be very helpful.
I have one question about JCache declarative configuration in hazelcast. I've read documentation but somehow I can't find an answer yet, so I'd much appreciate any help.
I'd like to configure my caches kind of dynamically (it applies mostly to hazelcast client but for server side probably should be the same). I need to join existing cluster (given hazelcast client/server instance with it own configuration) and then get cache manager with a declarative configuration of needed caches, something like this:
//server HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(hzConfig); CachingProvider caching = Caching.getCachingProvider("com.hazelcast.cache.impl.HazelcastServerCachingProvider"); Properties properties = new Properties(); properties.setProperty( HazelcastCachingProvider.HAZELCAST_INSTANCE_NAME, hazelcastInstance.getName()); CacheManager cacheManager = caching.getCacheManager(new URI("classpath:hz-my-cache-test.xml"), null, properties); Cache<Long, Object> cache = cacheManager.getCache("my-cache-name");
The same direction is for the client but with proper client caching provider, etc.
Is it possible at all?
Somehow from [THIS] (https://docs.hazelcast.org/docs/latest/manual/html-single/#scoping-to-join-clusters) hazelcast documentation I didn't get it and my prototype app doesn't read such configured caches. Although putting cache configuration directly into
hazelcast.xml works just fine.
Target of invocation cannot be found! Partition owner is null but partitions can't be assigned since all nodes in the cluster are lite members.