Redis client for Scala built on top of Cats Effect, Fs2 and Lettuce (formerly Fs2 Redis) -> https://redis4cats.profunktor.dev/
Hi folks, I have a question regarding atomicity. Actually first off shoutout to the maintainers, this is an awesome and well designed library.
I am trying to implement a "unique URL counter" in Redis for a school project, with multiple distributed actors updating the state. I am able to make it work easily by blindly using sAdd and finally sCard when querying for unique counts. But I wanted to know how I could accomplish this without the O(n) operation of pulling the cardinality upon query?
incr, but I had race conditions using applicative sequencing or even watch + transactions. Wondering if anyone has a tip for how to make things atomic when there is a set and a counter involved to make this a O(1) operation on write and query.
- key by "requests:" + url
- each key is a sortedset of ints
- values are epoch timestamps
- zrange lets you count how many requests happened in a time period
- you can use a method (I forget) to mass remove elements smaller than a value (clear stale values)
- add to the set
- return number of elements
Hi all. I'm a bit confused about the cluster connection APIs. When using Redis as a managed service, or deploying it via an operator or helm chart in K8s, the service is exposed by a single host-port pair e.g. redis-cluster:6379. Behind it are several nodes, usually with at least one shard each. What is the API for connecting to such a cluster? Doing something like:
uri <- Resource.liftF(RedisURI.make[IO]("redis://redis-cluster:6379"))
cli <- RedisClusterClient[IO](uri)
api <- Redis[IO].fromClusterClient(cli, RedisCodec.Utf8)results in a single connection that hops through DNS between the multiple nodes behind the exposed service URL. Ideally there should be 1 connection per Redis node, with automatic failover to the shards if they fail (which is done by Lettuce as far as I can tell).
All uris are tried in sequence for connecting initially to the cluster. If any uri is successful for connection, the others are not tried anymore.
@cosminci have a look at the docs: https://redis4cats.profunktor.dev/client.html#cluster-connection
I personally don't use the cluster support so pretty sure we could use some improvements if we get feedback from real users.
From lettuce
A connection object with Redis Cluster consists of multiple transport connections. These are:
Default connection object (Used for key-less commands and for Pub/Sub message publication
Connection per node (read/write connection to communicate with individual Cluster nodes)
When using ReadFrom: Read-only connection per read replica node (read-only connection to read data from read replicas)^ I'm thinking one of these hops between nodes while the other 2 are stable. What I'm seeing with a single pod / cluster client is in the screenshot above with the 3 switching to a different node every few seconds (but the interval is just an artifact of the prometheus scraping interval and grafana refresh interval).
evalSha for each request. Without the evalSha (no redis), the service handles ~ 24k RPS peak throughput, while with Redis, ~20k RPS. So a 17% performance impact overall, which is really great.
Lettuce connections require single-threaded/synchronized access when using transactions. If two or more threads call concurrently MULTI/EXEC methods this will destroy the connection state.
client.use { redis =>
val key = "mykey"
val tx = RedisTransaction(redis)
val commands = redis.incr(key) :: redis.expireAt(key, Instant.now()) :: HNil
val io = tx.filterExec(commands)
ZIO.collectAllParN_(10)(List.fill(100)(io)).catchAll(error => Task(println(error)))
}ERR MULTI calls can not be nested
@gvolpe it seems like redis4cats isn’t enabling any sort of topology refresh for Redis Cluster, possibly making the clients unstable whenever a failover occurs. The flatTap when initializing the lettuce client only retrieves an initial set of partitions. That flatTap could instead do something alone the lines of
client.setOptions(
ClusterClientOptions
.builder()
.topologyRefreshOptions(
ClusterTopologyRefreshOptions
.builder()
.enableAdaptiveRefreshTrigger(RefreshTrigger.MOVED_REDIRECT, RefreshTrigger.PERSISTENT_RECONNECTS)
.build()
)
.build()
)or even better, expose a way for users to set what kind of refresh mechanism they want according to https://github.com/lettuce-io/lettuce-core/wiki/Redis-Cluster#user-content-refreshing-the-cluster-topology-view. What do you think?
