by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 15:45
    seanpar203 commented #278
  • Jan 31 2019 15:45
    seanpar203 commented #278
  • Jan 29 2019 23:31
    ankitpatel96 commented #291
  • Jan 29 2019 21:39
    akrylysov commented #244
  • Jan 05 2019 08:42
    qianShaw commented #131
  • Jan 04 2019 13:18
    Grokzen commented #131
  • Jan 04 2019 03:35
    qianShaw commented #131
  • Jan 03 2019 14:41
    Grokzen closed #301
  • Jan 03 2019 14:41
    Grokzen commented #301
  • Jan 03 2019 14:04
    15astro commented #301
  • Jan 03 2019 00:04
    Grokzen commented #301
  • Jan 02 2019 17:54
    15astro edited #301
  • Jan 02 2019 17:47
    15astro opened #301
  • Dec 27 2018 15:17
    Grokzen commented #296
  • Dec 26 2018 17:16
    CheViana commented #296
  • Dec 26 2018 17:06
    CheViana commented #295
  • Dec 26 2018 13:10
    Grokzen synchronize #296
  • Dec 26 2018 13:10

    Grokzen on redis-py-3.0

    Ported in a few new testcases t… More unicode, bytestring fixes.… (compare)

  • Dec 25 2018 12:35
    coveralls commented #300
  • Dec 25 2018 12:35
    coveralls commented #300
John Loehrer
@72squared
in other news, I've been working to open-source some of the code I've been using on redis. Here's a package that is just getting off the ground: http://redpipe.readthedocs.io/en/latest/
hoping to get to a stable release soon.
Grokzen
@Grokzen
@72squared hey m8, long time no speak :p
we are right in the final weeks of the latest platform development so long days and long nights right now
John Loehrer
@72squared
hey I hear ya. No rush on this. If I get the itch, I'll work on the redis cluster scan support and submit a patch.
I've been fairly busy myself.
Grokzen
@Grokzen
cool code there with redpipe
have not looked into the details of it but looks very intrestegin
what is the main difference between tredpipe and regular pipelines in both redis-py and redis-py-cluster?
John Loehrer
@72squared
redpipe is a wrapper around the library that allows you to instantly get a reference object back from the method you call that you can use

so for example:

with redpipe.pipeline() as pipe:
    foo = pipe.incr('foo')
    bar = pipe.incr('bar)
    pipe.execute()
print([foo, bar])

which would print out:

[1, 1]
In this case, both foo and bar are Future objects that don't contain the actual data until pipe execute.
but it allows you to build functions that use those Futures inside a function .
this is because you can wrap one redpipe.pipeline() in another.
John Loehrer
@72squared
so I can build a class, where I pass in an optional redpipe and then do some work and assign those futures as class attributes. And that will allow me to chain that object encapsulation with other operations I can pipeline.
You can see all the concepts at work in Structs: http://redpipe.readthedocs.io/en/latest/structs.html#
read that page and then dig into how it does it, and you will see what I mean. But you can build your business logic around this, and still pass in pipeline objects to be able to combine multiple units of work together in a single pipeline operation.
The Keyspaces stuff just allow you to manipulate a set of keys by the id and ignore the convention of things like USER{A}, USER{B}, and instead just use A, and B as the key.
but it also allows you to specify a connection to a certain redis database, so you can actually manage multiple redis database connections transparently and still pipeline them.
John Loehrer
@72squared
that alone is no small feat.
It keeps track of all the calls bound for each database connection and then fires them off in different threads
John Loehrer
@72squared
The keyspace Hash objects also allow you to specify the encoding of the fields, and serialize it as json automatically, or convert to ints or floats, etc.
All of these concepts are borrowed from what I used to build marcopolo.me over the past few years. This is just a better version of it, because I got to re-architect it now that I know all the things I needed to build.
John Loehrer
@72squared
here's another spot where I explain that first concept I was trying to explain: http://redpipe.readthedocs.io/en/latest/redpipe.html#module-redpipe.pipelines
let me know if you have any questions. I am going to be giving a talk on all of this at redisconf at the end of the month. So I need to get better at explaining it. :-)
Grokzen
@Grokzen
ah cool that you are doing a talk about it, i hope videos is posted from them so i can watch later :D
i have not seen that many talks that even touches redis clustering so far
santosh2812
@santosh2812
Hi by using this ( https://github.com/Grokzen/docker-redis-cluster ) this code base and composing it to create redis cluster on docker, after adding some line related to PIP command in the Docker file it got executed. but Iam not able to connect with all the masters its only allow me to connect with 7002 port. I am using redis cluster for Session management in MVC with C#, it throws below exception when I changes the port in my config file it throws error- MOVED, But when I pass all the IP:PORT in the connection string it works and when 7002 port is removed from the list it again throws error ( have used RedisSessionStateStoreProvider) and for Microsoft.Web.Redis.RedisSessionStateProvider gives error for connection Multiplexer. it always says to increase the time, which does not work. Please suggest.
Grokzen
@Grokzen
@santosh2812 Hi. sorry for not responding sooner
First look if you are using the ports 7000-7007 to other services that prevents you from reserving them for redis cluster inside docker. The second you can check is if when you start the cluster that docker binds all ports for you or not on your host system. If not then you have to debug why that happens.
The other things i can comment on is that are you really using a cluster compatible client? The client must be capable of understanding the redis cluster spec and commands. You can't really use any old regular redis client and expect it to work in a cluster env
are you able to connect to all nodes in the docker cluster with the regular redis-cli client from outside the container?
enter your container and look if all redis-server processes is running and if they are using the correct ports
On the node that works, what do you get back when you run CLUSTER INFO command?
Grokzen
@Grokzen
I do think that you are using a redis client that is not compatible with a redis cluster and you should look into using a client that is for your usecase
santosh2812
@santosh2812
@Grokzen Hi, its ok. as the setup was on single container so and also was not able to figure out the problem So i started Sentinel configuration. but here also am not able to configure and understand the configuration I have some points please have a look.
As per the tutorial (https://github.com/AliyunContainerService/redis-cluster )the configuration is created with local IP of docker(172.17.0.2) when I do port forwarding in Docker file for the master and sentinel its available for connecting from local machine but the synchronization between the master and slave did not happen it throw below exception
Connecting to MASTER 192.168.80.169:6379
1:S 06 Jun 06:24:12.792 MASTER <-> SLAVE sync started
1:S 06 Jun 06:24:12.800
Non blocking connect for SYNC fired the event.
1:S 06 Jun 06:24:12.862 Master replied to PING, replication can continue...
1:S 06 Jun 06:24:12.869
Partial resynchronization not possible (no cached master)
1:S 06 Jun 06:24:12.870 * Master does not support PSYNC or is in error state (reply: -ERR Can't SYNC while not connected with my master).
Again to get the Master information we need to connect with Sentinel so it needs to be on port forwarding, when I specify the port forwarding on it then unable to scale it as we need more than 1 sentinel for electing the master in case of master failover. Here below exception is thrown
ERROR: for redissentinel_sentinel_3 Cannot start service sentinel: driver failed programming external connectivity on endpoint redissentinel_sentinel_3 (91803d790d07d933f9bfc7c2c431b23f8d633db37f6981705e1f758b07080c6a): Bind for 0.0.0.0:26379 failed: port is already allocated
Apart from above blockers there are lots of question coming as below which I have also posted on the forum but did not got any response on it, some of them are :
  1. How to expose the port 26379 by using sentinel announce-ip (my system Ip) to connect from outside.
  2. As we will get the master Information from The sentinels - which one to connect as I have 3.
  3. Do we need to right the logic to connect with the master in our application (MVC C#) to connect with the Details got form the point no 2. Or How.
  4. Do we need to expose the Slaves also? How.
Grokzen
@Grokzen
@santosh2812 Sentinel? I can't help you with sentinel setup, i have never used that setup. I have only used the built-in redis-cluster support inside redis. You probably have to turn to other more general redis channels to get help with sentinel setups
Another more general problem you appear to have is that you have problems with your port sharing from the docker container. There is not many things that can cause that
one main thing is that you either run multiple containers that expose the same port, run one container but try to expose the same port multiple times, or that you have redis installed/configured on your host machine and try to start a container with the same setup and they both want to use the sam port
the error port is already allocated is usually a indication or either point 1 or 3 from ^
Angus Pearson
@AngusP
This PR andymccurdy/redis-py#866 at redis-py might have some interesting implications for this cluster impl if merged
seems to make fairly big changes to the client methods, by adding support for namespacing functions. I haven't had a play with the code yet but this might be useful for auto hashtagging keys (hashtag as in https://redis.io/topics/cluster-spec#keys-hash-tags)
Grokzen
@Grokzen
@AngusP Is that not the third/fourth time that a namespace MR has been created? O.o
but still no merge yet
AlexBoyd
@AlexBoyd
@Grokzen Re: that earlier work I was doing to find a good way to support lazy initialization, we were able to solve it at the application layer.
We have a delegate object we use via something like this:
def getattr(self, attr):
if not self._redis:
self._redis = initializeRedisClient(self._clusterEnabled, self._config)
res = getattr(self._redis, attr)
return res
works a lot better than expected
without needing to touch the library code at all
I'm now looking to add something like your https://github.com/Grokzen/docker-redis-cluster docker container to our testing setup. Would it be possible for you to add on an MIT license to that repro? I'd love to use your work directly, but it always nice to be sure there is a license explicit.