Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jun 18 17:52
    mp911de labeled #1777
  • Jun 18 16:48
    christophstrobl opened #1777
  • Jun 18 09:49
    binoysankar commented #1776
  • Jun 18 08:52
    mp911de unlabeled #1776
  • Jun 18 08:52
    mp911de labeled #1776
  • Jun 18 08:52
    mp911de closed #1776
  • Jun 18 08:52
    mp911de commented #1776
  • Jun 18 08:50

    mp911de on 6.1.x

    Fix asynchronous resource clean… (compare)

  • Jun 18 08:50

    mp911de on main

    Fix asynchronous resource clean… (compare)

  • Jun 18 08:50

    mp911de on 6.0.x

    Fix asynchronous resource clean… (compare)

  • Jun 18 08:12
    binoysankar commented #1776
  • Jun 18 08:11
    binoysankar commented #1776
  • Jun 18 08:09
    binoysankar commented #1776
  • Jun 18 08:00
    binoysankar commented #1776
  • Jun 18 06:50
    mp911de labeled #1776
  • Jun 18 06:50
    mp911de commented #1776
  • Jun 18 04:28
    binoysankar edited #1776
  • Jun 18 01:04
    binoysankar opened #1776
  • Jun 17 14:16
    mp911de assigned #1774
  • Jun 17 10:29
    oridool commented #1774
denco
@denco:matrix.org
[m]
My assumption is that the try with resources block calling close of the pool will return the object back to the pool. Is that correct ?
Agustín Viera
@avierauy:matrix.org
[m]
Hello everyone! I have a question, what is 'withDatabase' of the client builder?
It's necessary to use?
tiagoedez
@tiagoedez
@avierauy:matrix.org this method configures the database number, please look https://lettuce.io/core/release/api/io/lettuce/core/RedisURI.Builder.html#withDatabase-int-
Mark Paluch
@mp911de
try (GenericObjectPool<StatefulRedisClusterConnection<String, String>> pool = lettuceConnector.getPool()) { closes the pool. It should actually be something like try(StatefulRedisClusterConnection<String, String> connection = pool.borrowObject()) {…}
Victor Babenko
@virusman
Is there an internal retry mechanism in Lettuce? We’re observing 20x traffic on the Redis node compared to what the app metrics report, and trying to figure out where the additional requests are coming from.
Mark Paluch
@mp911de
Yes, if a connection gets disconnected, then pending commands are retried.
Jaden Wright
@jaw.sonic_gitlab

Hi Mark/other lettuce experts, we've been testing lettuce to replace our jedis cache client as it seems to have some really promising improvements, but we've been having some issues with request latencies at the tail end of the distribution. Specifically, while our p99 latencies are under 5ms, p99.9 and especially p99.99 latencies were huge, up to 400ms consistently. We were able to pinpoint the cause as being the periodic topology refresher. Disabling periodic topology refresh brought p99.99 latency from 400ms to 40ms. However the periodic topology refresher seems to be the only way to properly recover from a failover, as the new replica node never gets picked up by lettuce, even with the adaptive refresh triggers. Extending the topology refresh period out longer helps but increases risk of running without lettuce's knowledge of topology changes.

Context:

  • application host count: ~100
  • redis (elasticache) cluster shards: ~40
  • one replica per master
  • traffic rate: ~5k req/s per host
  • using lettuce cluster client with sync api
  • using ReadFrom.ANY connections
  • using lettuce 6.0.1 release

Question:
I've seen you mention that this is a tricky problem without a perfect solution, and that periodic topology refresher does cause higher load. Is the effect we're seeing on p99.9 and p99.99 latency a necessary result of the periodic refresher, is that expected/known? I had thought that since its now non-blocking in lettuce 6 that it shouldn't affect the request latencies. Or is there some configuration setup we're doing wrong or something we're missing that can help solve this problem? I apologize if this chat room is not the right place for this sort of post.

Builder code (slightly approximate):

connectionTimeoutMillis = 65000; // // Also tested low values like 500ms
enableDynamicRefreshSources = false; // We found performance is better without this
enablePeriodicTopologyRefresh = true;
topologyRefreshPeriodSeconds = 60; // Also tested 30 seconds to 1 hour
autoReconnect = true;
requestQueueSize = 20; 
disconnectedBehavior = reject;
keepAlive = true;
tcpNoDelay = true;
pingBeforeActivateConnection = false; // Also tested true
commandTimeoutMillis = 50; // Higher timeout has lower timeout errors but higher p99.99 latency

RedisURI redisUri = RedisURI.Builder.redis(endpoint)
        .withPort(port)
        .withTimeout(Duration.ofMillis(connectionTimeoutMillis))
        .build();

RedisClusterClient client = createRedisClusterClient(redisUri);

ClusterTopologyRefreshOptions.Builder topologyRefreshOptionsBuilder = ClusterTopologyRefreshOptions.builder()
        .dynamicRefreshSources(enableDynamicRefreshSources)
        .enableAllAdaptiveRefreshTriggers();

// Default value of enable periodic refresh is false
if (enablePeriodicTopologyRefresh) {
    topologyRefreshOptionsBuilder.enablePeriodicRefresh(Duration.ofSeconds(topologyRefreshPeriodSeconds));
}

client.setOptions(ClusterClientOptions.builder()
        .autoReconnect(autoReconnect)
        .requestQueueSize(requestQueueSize)
        .disconnectedBehavior(disconnectedBehavior)
        .socketOptions(SocketOptions.builder()
                .connectTimeout(Duration.ofMillis(connectionTimeoutMillis))
                .keepAlive(keepAlive)
                .tcpNoDelay(tcpNoDelay)
                .build())
        .pingBeforeActivateConnection(pingBeforeActivateConnection)
        .topologyRefreshOptions(topologyRefreshOptionsBuilder.build())
        .timeoutOptions(TimeoutOptions.builder()
                .fixedTimeout(Duration.ofMillis(commandTimeoutMillis))
                .build())
        .build());
CasperLiu444
@CasperLiu444
Hey guys, I have a simple question, for standalone redis client, I couldn't find any API that reuse existing connections. I have a micro-service that supposed to accept requests and run Redis Transactions for every of them, I used RedisClient connect() method(https://lettuce.io/core/5.2.2.RELEASE/api/io/lettuce/core/RedisClient.html#connect--) to get connections, but it shows that it will create a new connection. Since I want it to constantly process requests, I want the application to just keep running even after that. this will just creates more and more connections. I don't know if I should close the connections every time it finished the redis transaction, or I should just reuse the existing connections if that's possible. Could you please tell me about the tradeoff there? Thank you so much
Victor Babenko
@virusman
@mp911de Does it print disconnects to the log by default? Is there a way to detect that?
I’m wondering if what might be happening is the node getting to 100%, being slow to respond, and Lettuce then reconnects and retries the whole pipeline.
Sanuj Bhadra
@sanuj96

Hey Guys,

I am using the Cluster mode disabled Redis of AWS Elasticache.

There is a reader endpoint (pointing to one replica) and a writer endpoint (pointing to primary always). How do I use the Lettuce library to connect to reader so that during maintaince/replacement there is no downtime/change in application ?
sravan
@securelyshare-sravan-nethi
Hello Guys,
getting below exception
Caused by: io.lettuce.core.RedisException: java.lang.IllegalArgumentException: Connection to X.X.X.X:6379 not allowed. This connection point is not known in the cluster view
at io.lettuce.core.cluster.PooledClusterConnectionProvider.getConnectionAsync(PooledClusterConnectionProvider.java:359)
at io.lettuce.core.cluster.ClusterDistributionChannelWriter.write(ClusterDistributionChannelWriter.java:93)
at io.lettuce.core.cluster.ClusterCommand.complete(ClusterCommand.java:60)
at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:558)
at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:511)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
any idea ?
Vikas Kumar
@vikas03:matrix.org
[m]
Hi Guys,
I was solving a bug where in in case redis is down after the application is launched and then an async write is done then i will not get any error like timeout, can someone guide me on how to handle this issue ?
Mark Paluch
@mp911de
@securelyshare-sravan-nethi check out the validateClusterNodeMembership setting in ClusterClientOptions, see https://lettuce.io/core/release/reference/index.html#clientoptions.cluster-specific-options
Kannadasan S
@Kannadasan89
@mp911de : Suddenly in a production environment, I am getting lot of info log printed [channel=0x6b1567b1, /10.x.x.x:43238 -> azure.redis.cache/12.x.x.x:6380, epid=0x3a9] Closing channel because endpoint is already closed But I am not sure, about the reason, Could you please suggest on this?
Mark Paluch
@mp911de
That is a potential race condition that gets caught. It can happen when previously the channel was disconnected and Lettuce tries to auto-reconnect but during the reconnect phase you close StatefulRedisConnection.
Once auto-reconnect was started, there's no way to interrupt it. To prevent connection/memory leaks, we close the newly created connection and log that event
Nothing really to worry about.
1 reply
Tobias Månsson
@tobias-zeptio
I'm trying to implement a client cache for a set, but can't find a way to do that. Is client caching only supported for get/set commands and string values?
I've created a CacheFrontend<String, List<String>>, but the smembers method is not available on this class.
vtczk
@vtczk

Hi guys,

Is it possible to implement command pipelineing using Reactive Api with Project Reactor ? If yes how it would be best approached? When should the flush operation be called?

Kannadasan S
@Kannadasan89
@mp911de : I have observed, In my springboot application, very frequently AUTH, CLIENT redis command is triggered by Lettuce for every N millisecs command. Is there a reason for that? Looks like it may impact an app performance. Application using azure redis cache with redis cluster enabled(premium)
Mark Paluch
@mp911de
That can happen if you don't have connection pooling enabled
2 replies
Mohammed Shehzad Malik
@shehzaDmaliKSR_twitter
Hi folks! I have one query about lettuce(java redis client), does lettuce support near cache(local cache) like Hazelcast(near cache)? If yes, would you please help me with some examples so that I can implement it as a PoC?
Mark Paluch
@mp911de
It doesn't because it is a Redis Client that communicates with Redis for each API call/command.
rs2152
@rs2152
Hello Guys, I'm facing bad experience with pub/sub with redis, I'm using Lettuce connection, it is right way, or it is problem with connection pooling with redis?
Tyler Reid
@twreid
I have a question about the mget command against a redis cluster and how timeouts work for it. I have been investigating an issue where we have anywhere from 5 - 20% of our redis calls are timing out and I was just curious how the timeout works for mget and if the timeout applies to the whole call or if it gets split when different calls are made to different shards. I haven't had a chance to deploy the service with debug logs yet to get that information, but if I understand how the timeouts are applied for mget it might help me a bit too
For reference this is how we make the call List<KeyValue<String, String>> response = commands.mget(keys); commands is a RedisAdvancedClusterCommands<String, String> and keys can vary by incoming request anywhere from 1 - 250 with a typical average of around 20 - 30 that is why I am curious how the timeouts work for this call
Mark Paluch
@mp911de
Each multi-key command requires the key to map to the same slot when using Redis Cluster. Lettuce can lift this restriction by inspecting keys and by partitioning the call into multiple MGET commands and merge results after the call
Tyler Reid
@twreid
Ok thank you. I have done some more investigation and I think that is what is happening to me and for some of the larger requests it get's broken down into several commands and some of them timeout before they get a chance to execute. I'm looking at different ways I can fix this, but may end up increasing the timeout a bit.
Jaden Wright
@jaw.sonic_gitlab
Hi there, just wanted to check in again - is it expected that some small percentage of commands take significantly longer to complete in Lettuce? That is, 99% of commands are replied within 5ms, but less than 1% can take 50-500ms?
Lazar Bulić
@pendula95
I have trouble wrapping my head around this. I am using Lettuce Reactive Cluster Client with set PeriodicRefresh. Our cluster (16 master and 16 slaves) is split to 2 Virtual Machines with failover enabled, which means the Redis cluster will recover if one of the machines fails. In our app I am using CLUSTER INFO command and extracting cluster_state:ok as live check for my app. So when the failover process is done and now cluster is fully on one VM my CLUSTER INFO command always timeouts (I have setup a command timeout) as it is always sending the command to the redis server which is now dead (unreachable). Why is CLUSTER INFO ignoring the new topology obtain from period refresh?
Yair Lenga
@yairlenga
Greetings.
I'm trying to use the Redis 6 new feature that allow sub-second timeout on blocking operation (BLPOP and friends). However, current Lettuce client code only take integer as timeout, forcing me for minimum timeout of 1 sec. Is there a way, with current code to get sub-second timeout ? Also, what is the proper place the raise this issue - so that it will be addressed in future update/fixes. My current solution is to use Lua, but Lua script have limited ability to block at all, so implementation is very inefficiient/cumbersome.
joolskii
@joolskii
Is anyone else having an issue getting access to the documentation on the lettus.io site ? Seems to be a cloudflare issue....
Mark Paluch
@mp911de
It's fixed now @joolskii. Sorry for inconvenience.
joolskii
@joolskii
No problem, thanks for fixing it 👍
mrbublos
@mrbublos
hi all, can anyone share why socketOptions.keepAlive is disabled by default? isn't it always a good thing to have?
Hasan Abbasi
@redviper
Has anyone got a sample or some high level overview about using a custom topology refresh to avoid creating new connections for each refresh? We have 3000 client nodes and the constant barrage of new connections drives the tail latencies up (and occasional timeouts also)
chankaya-singh
@binarypirate:matrix.org
[m]
@mp911de: We are using lettuce as our redis client, we are observing redis command timeout in our application log, we increased our redis timeout to 5 min and we have reduce this occurrence to 95 % but still sometime we get this exception in our logs.
Mark Paluch
@mp911de
Tech Enthusiast
@suryamegham1729
fundamental question: what happens I have run a command for "watch" a key and the app got exception thread closed, didn't execute the exec/unwatch command? any side effects? will the key "watched" indefinitely?
@mp911de
Mark Paluch
@mp911de
Redis tracks WATCH for a connection. So if you reconnect, then the WATCH is gone. Other than that, if WATCH isn't followed by MULTI, then it remains active indefinitely I think
Tech Enthusiast
@suryamegham1729
Thanks @mp911de :)
Karin-Aleksandra Monoid
@paranoidmonoid
Hello. In my Kotlin app, I wanted to keep track on key expiration in Redis via a listener and based on that event I wanted to call some suspend functions. The thing is, the listener interface has no suspend function(s) (I refer to@FunctionalInterface public interface PushListener and its method void onPushMessage(PushMessage message); respectively) and I have to run everything in runBlocking . Is there any ways to avoid blocking calls or any plans to support suspendable "onMessage" in the future?