scan(…) usage requires to be called in a loop with the outcome of the previous cursor. This can be a bit tricky in async/reactive arrangements so we decided to provide io.lettuce.core.ScanStream which performs incremental scans behind the scenes while you consume the scan result as Flux<K>.
io.lettuce.core.protocol.TransactionalCommand instances held within io.lettuce.core.StatefulRedisConnectionImpl.multi keeps on growing until the JVM crashes. I'm sure this is not a bug and my misconfiguration but unable to resolve it.
3 replies
According to the Redis docs:
It is possible for an infinite number of clients to iterate the same collection at the same time, as the full state of the iterator is in the cursor, that is obtained and returned to the client at every call. Server side no state is taken at all.
isFinished). Apologies for the noise!
Hello, I have a setup involving a Spring Boot microservice using Spring Data Redis, Atomikos for transactions and Lettuce for Redis. I use a standalone Redis and make use of transactions in my application. I'm trying to identify the root cause of a memory leak in Lettuce when Redis goes down. The queue of
io.lettuce.core.protocol.TransactionalCommandinstances held withinio.lettuce.core.StatefulRedisConnectionImpl.multikeeps on growing until the JVM crashes. I'm sure this is not a bug and my misconfiguration but unable to resolve it.
@mp911de I have a simple application that can reproduce this now - https://github.com/radarsh/lettuce-oom.
TransactionalCommand, it will show that it will only increase now.
ClientOptions (specifically disconnectedBehavior and requestQueueSize), see https://lettuce.io/core/release/reference/index.html#client-options
@Bean
public LettuceConnectionFactory redisConnectionFactory() {
RedisStandaloneConfiguration redisConfiguration = new RedisStandaloneConfiguration();
redisConfiguration.setHostName(hostName);
redisConfiguration.setPort(port);
redisConfiguration.setPassword(password);
redisConfiguration.setDatabase(databaseIndex);
//Redis Cluster configuration
GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
poolConfig.setMaxIdle(MAX_IDLE);
poolConfig.setMaxTotal(MAX_TOTAL);
LettuceClientConfiguration lettuceClientConfiguration=LettucePoolingClientConfiguration.builder().
poolConfig(poolConfig)
.clientOptions(clientOptions())
.clientResources(clientResources())
.build();
LettuceConnectionFactory lettuceConnectionFactory=new LettuceConnectionFactory(redisConfiguration, lettuceClientConfiguration);
lettuceConnectionFactory.setValidateConnection(true);
lettuceConnectionFactory.afterPropertiesSet();
return lettuceConnectionFactory; }
@Bean
public ClientOptions clientOptions(){
return ClientOptions.builder()
.disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
.pingBeforeActivateConnection(true)
.autoReconnect(true)
.build();
}
@Bean(destroyMethod = "shutdown")
ClientResources clientResources() {
return DefaultClientResources.create();
}
I see BLPOP command is written with 0
This is correct.
TimeoutSourceconsiders only the timeout for command synchronization (how long to wait for a command to complete)
I try to set TimeoutConfig but it also didn’t work, and it's timeoutCommands disabled by defalut. But I found this reply by you.Maybe I have to learn the best practice in redisTempalteand lettuce at first.thks
(error) NOAUTH Authentication required. OR wrong password) at the client side, does Lettuce provide support for that?
Hi, I am observing a strange issue with "RedisClusterClient" and connection events.
My redis server has an idle timeout of 2 mins. As I mentioned, I am using "RedisClusterClient". I have also added an eventBus listener. In my sample code, I am doing a keepalive every 30secs (by writing to a dummy entry). But, despite this, i receive "ConnectionDeactivated" event every 2 minutes (followed by connection activated). It doesn't matter if i use async() or sync() APIs, I receive the disconnect events every 2 mins.
But, If I use "RedisClient", i.e. the regular, non-cluster client, I do not see this behaviour - I do not see any disconnect events since I am doing a keepalive every 30secs.
Sample code:
RedisClusterClient redisClient = RedisClusterClient.create("redis://localhost:6379");
StatefulRedisClusterConnection<String, String> connection = redisClient.connect();
RedisAdvancedClusterAsyncCommands<String, String> handle = connection.async();
TimerTask batchWriterTask = new TimerTask() {
@Override
public void run() {
handle.set("keepalive", "keepalive");
System.out.println("keepalive");
}
};
new Timer("keepalive").scheduleAtFixedRate(batchWriterTask, 0, 30000);
redisClient.getResources().eventBus().get().subscribe((event) -> {
System.out.println(event);
});
while (true);It would be great if I get some clarification. Thank you!
2020-05-29 05:59:29.572 PDT [keepalive] INFO c.a.a.a.LettuceSampleCluster - keepalive
.
.
.
.
2020-05-29 05:59:59.613 PDT [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.RedisStateMachine - Decoded ClusterCommand [command=AsyncCommand [type=SET, output=StatusOutput [output=OK, error='null'], commandType=io.lettuce.core.protocol.Command], redirections=0, maxRedirections=5], empty stack: true
2020-05-29 06:00:29.570 PDT [keepalive] DEBUG i.l.core.protocol.DefaultEndpoint - [channel=0xe23f8cee, /10.1.4.105:55398 -> /10.164.0.51:6379, epid=0x3] write() writeAndFlush command ClusterCommand [command=AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], redirections=0, maxRedirections=5]
2020-05-29 06:00:29.571 PDT [keepalive] DEBUG i.l.core.protocol.DefaultEndpoint - [channel=0xe23f8cee, /10.1.4.105:55398 -> /10.164.0.51:6379, epid=0x3] write() done
2020-05-29 06:00:29.571 PDT [keepalive] INFO c.a.a.a.LettuceSampleCluster - keepalive
2020-05-29 06:00:29.571 PDT [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.CommandHandler - [channel=0xe23f8cee, /10.1.4.105:55398 -> /10.164.0.51:6379, chid=0x3] write(ctx, ClusterCommand [command=AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], redirections=0, maxRedirections=5], promise)
2020-05-29 06:00:29.572 PDT [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.CommandEncoder - [channel=0xe23f8cee, /10.1.4.105:55398 -> /10.164.0.51:6379] writing command ClusterCommand [command=AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], redirections=0, maxRedirections=5]
2020-05-29 06:00:29.612 PDT [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.CommandHandler - [channel=0xe23f8cee, /10.1.4.105:55398 -> /10.164.0.51:6379, chid=0x3] Received: 5 bytes, 1 commands in the stack
2020-05-29 06:00:29.612 PDT [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.CommandHandler - [channel=0xe23f8cee, /10.1.4.105:55398 -> /10.164.0.51:6379, chid=0x3] Stack contains: 1 commands
2020-05-29 06:00:29.612 PDT [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.RedisStateMachine - Decode ClusterCommand [command=AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], redirections=0, maxRedirections=5]
2020-05-29 06:00:29.612 PDT [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.RedisStateMachine - Decoded ClusterCommand [command=AsyncCommand [type=SET, output=StatusOutput [output=OK, error='null'], commandType=io.lettuce.core.protocol.Command], redirections=0, maxRedirections=5], empty stack: true
DisconnectedEvent [/10.1.4.105:55396 -> /10.164.0.51:6379]
2020-05-29 06:00:30.042 PDT [lettuce-nioEventLoop-4-2] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0xafd7dac9, /10.1.4.105:55396 -> /10.164.0.51:6379, last known addr=/10.164.0.51:6379] channelInactive()
2020-05-29 06:00:30.043 PDT [lettuce-nioEventLoop-4-2] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0xafd7dac9, /10.1.4.105:55396 -> /10.164.0.51:6379, last known addr=/10.164.0.51:6379] scheduleReconnect()
2020-05-29 06:00:30.043 PDT [lettuce-nioEventLoop-4-2] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0xafd7dac9, /10.1.4.105:55396 -> /10.164.0.51:6379, last known addr=/10.164.0.51:6379] Reconnect attempt 1, delay 1ms
ConnectionDeactivatedEvent [/10.1.4.105:55396 -> /10.164.0.51:6379]
2020-05-29 06:00:30.152 PDT [lettuce-eventExecutorLoop-1-2] INFO i.l.core.protocol.ConnectionWatchdog - Reconnecting, last destination was /10.164.0.51:6379
2020-05-29 06:00:30.155 PDT [lettuce-eventExecutorLoop-1-2] DEBUG i.l.c.c.RoundRobinSocketAddressSupplier - Resolved SocketAddress 10.164.0.51:6379 using for Cluster node f89444b6339b454aaaa99f2fb6405ddb01889ac4
2020-05-29 06:00:30.155 PDT [lettuce-eventExecutorLoop-1-2] DEBUG i.l.c.protocol.ReconnectionHandler - Reconnecting to Redis at 10.164.0.51:6379
2020-05-29 06:00:30.174 PDT [lettuce-nioEventLoop-4-4] DEBUG i.l.core.protocol.CommandHandler - [channel=0x49aa6237, [id: 0x49aa6237] (inactive), chid=0x4] channelRegistered()
ConnectedEvent [/10.1.4.105:55426 -> /10.164.0.51:6379]
2020-05-29 06:00:30.216 PDT [lettuce-nioEventLoop-4-4] DEBUG i.l.core.protocol.CommandHandler - [channel=0x49aa6237, /10.1.4.105:55426 -> /10.164.0.51:6379, chid=0x4] channelActive()
2020-05-29 06:00:30.216 PDT [lettuce-nioEventLoop-4-4] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0x49aa6237, /10.1.4.105:55426 -> /10.164.0.51:6379, last known addr=/10.164.0.51:6379] channelActive()
2020-05-29 06:00:30.216 PDT [lettuce-nioEventLoop-4-4] DEBUG i.l.core.protocol.CommandHandler - [channel=0x49aa6237, /10.1.4.105:55426 -> /10.164.0.51:6379, chid=0x4] channelActive() done
2020-05-29 06:00:30.217 PDT [lettuce-nioEventLoop-4-4] INFO i.l.c.protocol.ReconnectionHandler - Reconnected to 10.164.0.51:6379, Channel channel=0x49aa6237, /10.1.4.105:55426 -> /10.164.0.51:6379
ConnectionActivatedEvent [/10.1.4.105:55426 -> /10.164.0.51:6379]
2020-05-29 06:00:30.217 PDT [lettuce-nioEventLoop-4-4] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0x49aa6237, /10.1.4.105:55426 -> /10.164.0.51:6379, last known addr=/10.164.0.51:6379] userEventTriggered(ctx, io.lettuce.core.ConnectionEvents$Activated@20491923)
2020-05-29 06:00:59.570 PDT [keepalive] DEBUG i.l.core.protocol.DefaultEndpoint - [channel=0xe23f8cee, /10.1.4.105:55398 -> /10.164.0.51:6379, epid=0x3] write() writeAndFlush command ClusterCommand [command=AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], redirections=0, maxRedirections=5]
2020-05-29 06:00:59.571 PDT [keepalive] DEBUG i.l.core.protocol.DefaultEndpoint - [channel=0xe23f8cee, /10.1.4.105:55398 -> /10.164.0.51:6379, epid=0x3] write() done
2020-05-29 06:00:59.571 PDT [keepalive] INFO c.a.a.a.LettuceSampleCluster - keepalive
.
.
Just for an expt, i configured the redis server timeout to be 0. I observed no disconnections (as expected, I believe)
And, just to reiterate, I don't see the 2min disconnects with "RedisClient" - this is seen only with "RedisClusterClient"
RedisClusterClient creates multiple connections (a default connection for key-less commands and a connection for each shard for command routing). So even when sending a keep-alive on one connection, the other connection(s) in RedisClusterClient do not get kept alive.
1 reply
Could someone point me a to a link which advises on how we could rollout AUTH or change AUTH password for a production redis cluster, if we have to handle the failure scenarios arising due to the rollout like (
(error) NOAUTH Authentication required.OR wrong password) at the client side, does Lettuce provide support for that?
could someone give their guidance on this?