Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Oct 20 20:37
    mp911de unlabeled #1466
  • Oct 20 20:37
    mp911de labeled #1466
  • Oct 20 19:31
    RossRosem closed #1466
  • Oct 20 19:31
    RossRosem commented #1466
  • Oct 20 18:39
    mp911de commented #1466
  • Oct 20 16:27
    mashrafi29 commented #1463
  • Oct 20 15:26
    mp911de edited #1467
  • Oct 20 15:26
    mp911de edited #1467
  • Oct 20 15:25
    mp911de pinned #1467
  • Oct 20 15:25
    mp911de labeled #1467
  • Oct 20 15:25
    mp911de opened #1467
  • Oct 20 14:16
    RossRosem commented #1466
  • Oct 20 13:46
    mp911de commented #1466
  • Oct 20 13:18
    RossRosem commented #1466
  • Oct 20 13:14
    RossRosem reopened #1466
  • Oct 20 06:51
    mp911de commented #1465
  • Oct 20 06:50
    mp911de commented #1466
  • Oct 20 03:40
    RossRosem closed #1466
  • Oct 20 03:40
    RossRosem labeled #1466
  • Oct 20 03:40
    RossRosem opened #1466
Mark Paluch
@mp911de
Do you know whether there are AWS-internal events that an application could react to, when a failover occurs?
That would be helpful if an app could listen to something and then tell Lettuce to refresh the topology so that apps can be more resilient
Kosui Iwasa
@iwasa-kosui

whether there are AWS-internal events that an application could react to, when a failover occurs

I don't know, but with the discussion, I think the answer is no.

However, I think that the situation becomes better with enabling a periodic refresh.
ALSowjanya
@ALSowjanya
In our testing phase, we have seen that channels are closed, while the failover happens on AWS elasticache. Hence we have adaptive refresh enable and not periodic refresh, which helped in getting the cluster nodes details and re establishing the connection with new master and replica
Kosui Iwasa
@iwasa-kosui
It sounds good, thank you. Did you use StatefulRedisMasterReplicaConnection?
ALSowjanya
@ALSowjanya
We have used StatefulRedisClusterConnection, however we havent tested the scenario where there are no commands executed during the failover. And also we havent accounted if the reads are performed by master or replica, we have concentrated on if the commands were getting processed or not
Kosui Iwasa
@iwasa-kosui
Okay, thank you very much.
Lukáš Vasek
@bilak
Hello, sorry for writing here as it is related to spring and caching also, but I'm trying to solve the issue and find the best way. here is some description of my issue. Basically its simple spring application using default configuration for caching using redis. It's deployed on azure and using azure's redis chache. There's only one method annotated with @Cacheable. I'm randomly seeing connetcion reest by peer issues and then this issue is propagated to client as 500. I'm wondering whether lettuce cuould be somehow configured to do a retry on this type of error (some exponencial backoff would be perfect) or I need to do it on my own wrapping cached method in some try catch block.
Mark Paluch
@mp911de
connection reset by peer is something that Lettuce can't do anything about since it's a remote signal. It's propagated to let the caller know that a command might be executed but we don't know (either transmission or command response reading failed)
Lukáš Vasek
@bilak
@mp911de and how are you handling such cases in your applications? Do you some retry on such exception or you just return 500 to user?
Mark Paluch
@mp911de
It depends on the application. Retry is on idempotent operations is fine, but non-idempotent (e.g. SET to a shared key, INCR) can be risky.
Specifically, CommandHandler.exceptionCaught performs the exception propagation into the head command which is currently active.
You can customize that behavior in the sense of registering a ChannelInboundHandler that handles the exception before CommandHandler does.
You can register own handlers via NettyCustomizer.afterChannelInitialized(…) in ClientResources
Lukáš Vasek
@bilak
Do you know if I can do that somehow in spring way e.g. with some customization of lettuce? I think netty is too low level for me, but I can look into it
I was looking into class LettuceConnectionConfiguration so is there any possibility to change something?
Mark Paluch
@mp911de
No, there's no setting in Lettuce.
Regarding Spring Boot, let me check whether there's a ClientResources bean
Yeah, Boot registers ClientResources
So you would register an own ClientResources bean with a bit of customization
The code should look like https://gist.github.com/mp911de/85107508aa8391a9b862caffa5fd552b
Ideally, IOExceptions are followed by a disconnect signal. Lettuce auto-reconnects by default and so all commands that were sent and that didn't time out will be retried.
Lukáš Vasek
@bilak
That's what I thought that it's reconnected automatically and I would not see any issue on client, but it's not like that.
Ok anyway thank you for hints
Mark Paluch
@mp911de
Let us know if you need additional guidance and whether the suggestion has worked for you.
Lukáš Vasek
@bilak
I would be glad for any help, but the biggest issue is that I don't know how to simulate that issue because it's very random and I don't know whethe there is some network issue or redis is cutting of the connections :/
Mark Paluch
@mp911de
You could kill the process with kill -9. You can try leaving the process crashed or restart it immediately. Using a long-running command such as BRPOP could be also useful
Maybe even sending another command while BRPOP was sent and the server was killed
Lukáš Vasek
@bilak
@mp911de I wasn't trying your solution yet, but I have one idea. Please just let me know whether that could solve my issue or I'm on wrong path. I've found in LettuceConnectionFactory that it's possible to set validateConnection. So if I add a pool to lettuce and somehow set the validateConnection=true then each time then connection is being used it's validated before. So if there is such connection reset by peer is it possible that this won't be propagated to client? I know that there are going to be 2 round trips (validation + actuall call).
Mark Paluch
@mp911de
That should be fine, too.
Lukáš Vasek
@bilak
only thing that I don't know how to easily set is validateConnection. So with current spring implementation it's only possible to create new LettuceConnetcionFactory right? There is no customizer for that.
Mark Paluch
@mp911de
Yes, I think so
Alternatively, use a BeanPostProcessor
Lukas SP
@Lukaesebrot
Hey. I get a io.lettuce.core.RedisException: java.lang.NoSuchMethodError: 'void io.netty.util.concurrent.PromiseCombiner.<init>(io.netty.util.concurrent.EventExecutor)' everytime I close my RedisClient and I think this is because I have a piece of software which already uses Netty and so already defined it. Now I don't know how to fix this, as it seems like both use different versions. Can someone help me out?
Lukas SP
@Lukaesebrot
I fixed it by forking lettuce and relocating the netty dependency.
Lukáš Vasek
@bilak
and wouldn't it be easier in you project just use some kind of dependency management and override the version there?
mohanrao
@mohanrao
I am trying to connect to redis cluster then i am getting ther error as io.lettuce.core.cluster.RedisClusterClient.lambda$connect$13():620 - No subject alternative names matching IP address 10.25.17.135 found
@mp911de I cannot modify the certificate because i am using azure hosted service
Is there any example how to do the SocketAddressResolver fix
Mark Paluch
@mp911de
You can disable peer verification @mohanrao. Other than that, we plan to improve on host verification options with lettuce-io/lettuce-core#1460
Scott Duffey
@scottrduffey_twitter
Hello. I have a Spring application I am running on Amazon ECS, and it is failing to boot with Caused by: io.lettuce.core.RedisCommandExecutionException: ERR unknown command `CONFIG`, with args beginning with: `GET`, `notify-keyspace-events`,. Is this related to spring-projects/spring-session#124 ? I have found no explicit references to Spring Session or Lettuce in the code.
Scott Duffey
@scottrduffey_twitter
I found references to Spring Session in the code and learned Lettuce is the default Redis driver. The solution in the above linked issue is lengthy, and I'm reluctant to apply it because I don't understand why Redis doesn't work by default.
Mark Paluch
@mp911de
Spring Data Redis tries to auto-configure keyspace events to KEA
The issue is related to the linked ticket
Switching to Jedis will give you the same exception but with a Jedis exception type
The reason of the exception is that AWS has disabled the CONFIG and Spring Session tries to update the Redis server to enable keyspace events.
Marcus Rodan
@Nerja
Hi, I'm trying to enable cluster topology refresh with a period of 10s on Lettuce version 5.3.3. It works as expected as long as I don't use the option publishOnScheduler(true). I, however, start to see a lot of exceptions like "java.util.concurrent.RejectedExecutionException: event executor terminated" if I enable publishOnScheduler(true). Any ideas on what could be the cause of this issue? Thanks in advance, for any help.
1 reply
Andrew Lundgren
@ablundgren
I'm trying to tune my timeouts down to avoid seeing a now dreaded: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:119)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:131)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:79)
at com.sun.proxy.$Proxy127.lrange(Unknown Source
I am using: io.lettuce:lettuce-core:jar:5.3.4.RELEASE:compile
I am using redis cluster in AWS.
I have set the ClusterClientOptions.... timeoutOptions( to 100ms)
And the StatefulRedisClusterConnection connection timeout to 100ms. Where is the 1 minute timeout coming from
1 reply
ek-ex
@ek-ex
Hi all, I'm new here and hoping I can get your thoughts. Setting ReadFrom.ANY is only reading from master node. I would like a way to read from any and all nodes in the cluster. any idea how to achieve this?