Lettuce/Lobby

Hi Mark/other lettuce experts, we've been testing lettuce to replace our jedis cache client as it seems to have some really promising improvements, but we've been having some issues with request latencies at the tail end of the distribution. Specifically, while our p99 latencies are under 5ms, p99.9 and especially p99.99 latencies were huge, up to 400ms consistently. We were able to pinpoint the cause as being the periodic topology refresher. Disabling periodic topology refresh brought p99.99 latency from 400ms to 40ms. However the periodic topology refresher seems to be the only way to properly recover from a failover, as the new replica node never gets picked up by lettuce, even with the adaptive refresh triggers. Extending the topology refresh period out longer helps but increases risk of running without lettuce's knowledge of topology changes.

Context:

• application host count: ~100
• redis (elasticache) cluster shards: ~40
• one replica per master
• traffic rate: ~5k req/s per host
• using lettuce cluster client with sync api
• using ReadFrom.ANY connections
• using lettuce 6.0.1 release

Question:
I've seen you mention that this is a tricky problem without a perfect solution, and that periodic topology refresher does cause higher load. Is the effect we're seeing on p99.9 and p99.99 latency a necessary result of the periodic refresher, is that expected/known? I had thought that since its now non-blocking in lettuce 6 that it shouldn't affect the request latencies. Or is there some configuration setup we're doing wrong or something we're missing that can help solve this problem? I apologize if this chat room is not the right place for this sort of post.

Builder code (slightly approximate):

connectionTimeoutMillis = 65000; // // Also tested low values like 500ms
enableDynamicRefreshSources = false; // We found performance is better without this
enablePeriodicTopologyRefresh = true;
topologyRefreshPeriodSeconds = 60; // Also tested 30 seconds to 1 hour
autoReconnect = true;
requestQueueSize = 20; 
disconnectedBehavior = reject;
keepAlive = true;
tcpNoDelay = true;
pingBeforeActivateConnection = false; // Also tested true
commandTimeoutMillis = 50; // Higher timeout has lower timeout errors but higher p99.99 latency

RedisURI redisUri = RedisURI.Builder.redis(endpoint)
        .withPort(port)
        .withTimeout(Duration.ofMillis(connectionTimeoutMillis))
        .build();

RedisClusterClient client = createRedisClusterClient(redisUri);

ClusterTopologyRefreshOptions.Builder topologyRefreshOptionsBuilder = ClusterTopologyRefreshOptions.builder()
        .dynamicRefreshSources(enableDynamicRefreshSources)
        .enableAllAdaptiveRefreshTriggers();

// Default value of enable periodic refresh is false
if (enablePeriodicTopologyRefresh) {
    topologyRefreshOptionsBuilder.enablePeriodicRefresh(Duration.ofSeconds(topologyRefreshPeriodSeconds));
}

client.setOptions(ClusterClientOptions.builder()
        .autoReconnect(autoReconnect)
        .requestQueueSize(requestQueueSize)
        .disconnectedBehavior(disconnectedBehavior)
        .socketOptions(SocketOptions.builder()
                .connectTimeout(Duration.ofMillis(connectionTimeoutMillis))
                .keepAlive(keepAlive)
                .tcpNoDelay(tcpNoDelay)
                .build())
        .pingBeforeActivateConnection(pingBeforeActivateConnection)
        .topologyRefreshOptions(topologyRefreshOptionsBuilder.build())
        .timeoutOptions(TimeoutOptions.builder()
                .fixedTimeout(Duration.ofMillis(commandTimeoutMillis))
                .build())
        .build());