by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 21 23:01
    bakwc edited #117
  • Sep 18 07:56
    alexeymyltsev closed #121
  • Sep 18 07:56
    alexeymyltsev commented #121
  • Sep 17 12:11
    alexeymyltsev commented #121
  • Sep 17 12:05
    bakwc commented #121
  • Sep 17 12:03

    bakwc on 0.3.7

    (compare)

  • Sep 17 12:02

    bakwc on master

    New version (compare)

  • Sep 17 12:00
    bakwc commented #121
  • Sep 17 11:58
    alexeymyltsev commented #121
  • Sep 17 11:39
    bakwc closed #122
  • Sep 17 11:39
    bakwc commented #122
  • Sep 17 11:38

    bakwc on master

    Ignore self address if it exist… (compare)

  • Sep 17 11:38
    bakwc closed #124
  • Sep 17 11:38
    bakwc opened #124
  • Sep 17 11:38

    bakwc on allowSelfAddrInOtherAddrs

    Ignore self address if it exist… (compare)

  • Sep 17 11:31
    coveralls commented #123
  • Sep 17 11:27

    bakwc on master

    Fixed readonly nodes disconnect… (compare)

  • Sep 17 11:27
    bakwc closed #123
  • Sep 17 11:22
    bakwc opened #123
  • Sep 17 11:22

    bakwc on fixedReadonlyNodesException

    Fixed readonly nodes disconnect… (compare)

Matt Hall
@mattall

Hey all, just looking for some pointers getting started with syncobj. First, If I run the example 'kvstorage.py' how can get multiple servers running? I ran 'python kvstorage.py 127.0.0.1:9999 127.0.0.2:9999 127.0.0.3:9999'. Do I have to run a similar command in another terminal with the self address and partner addresses change? When I tried doing that I got an error "OSError: [Errno 49] Can't assign requested address"

Secondly, after running the first command (python kvstorage.py 127.0.0.1:9999 127.0.0.2:9999 127.0.0.3:9999') I tried to do ">>> set name 'matt'" but "get name" returned "None".

Thanks for any pointers. I'm really excited to start using SyncObj for my project if I can just get these little things figured out.

Filipp Ozinov
@bakwc
Hi, first - you should use different ports, not ip. You have only one local ip, 127.0.0.1, there is no ip 127.0.0.2 or 127.0.0.3. And you should change the first argument (self address), cause you cant have the same self address for two apps. So in terminal1 you should run: kvstorage.py 127.0.0.1:9999 127.0.0.1:9998 127.0.0.1:9997, in terminal2 kvstorage.py 127.0.0.1:9998 127.0.0.1:9997 127.0.0.1:9999 and in terminal3 kvstorage.py 127.0.0.1:9997 127.0.0.1:9999 127.0.0.1:9998
Second - for 3-node cluster you need at least two nodes up. In you case no connection was established with other nodes.
Matt Hall
@mattall
Thanks, that helps. I'm trying to use this for my client server app. How can I define a handle() method in kvstorage to use the tcp server that SyncObj creates? I tried modifying kvstorage-httpserver to work for TCP connections but get an "Address already in use" error. Instead of httpServer = HTTPServer(('', httpPort), KVRequestHandler) I'm doing TCPServer = socketserver.TCPServer((selfHost,int(selfPort)), KVRRequestHandler)If I can get a generic version of this to work I'll gladly make a pull request to add it as en example, but there's also a chance that I'm just approaching this the completely wrong way.
Filipp Ozinov
@bakwc
You can't have two tcp servers on the same port. You need to use another port for TCPServer, (selfHost,int(selfPort) is already used for SyncObj.
Matt Hall
@mattall
The TcpServer for SyncObj doesn't have a handler. Can I write my own handler for it, or would it be easier to just create a new server on a new port to let clients interact with the cluster from there?
Filipp Ozinov
@bakwc
It would be easier to create a new server on a new port for clients.
AmithGopal
@amith1893
How do I stop just one of the servers out of all the servers running ?
Matt Hall
@mattall
kill -9 <procid>
AmithGopal
@amith1893
Thanks. @mattall I want to kill the leader of the process cluster
*server
How do I find which server is the leader ?
Matt Hall
@mattall
you can call getStatus, a SyncObj method. This will give you info about the cluster. You can also call _getLeader
AmithGopal
@amith1893
Thanks @mattall
AmithGopal
@amith1893
how to make any function call on the Class Derived from SyncObject blocking ?
Filipp Ozinov
@bakwc
functions marked by @replicated decorator blocking by default. Or you can pass argument sync=True, and no callback argument.
batkins61
@batkins61
I have an application where multiple nodes receive an event, and one of them should then send out a notification. My current plan is to keep what notifications have already been sent in a ReplDict and use leadership to determine which node actually sends the notification. However, I see comments in the code that warn that there could be multiple leaders. I'm trying to understand how this can come about in this implementation and how to defend against it. Any other advice is most welcome as well.
Filipp Ozinov
@bakwc
This is possible due to raft protocol limitations. There is no guarantee that there is only one leader at any time moment. For example, when your cluster is splitted into several parts due to some network issues, a part with the current leader will still think that leader has not changed, and another part can elect another leader.
You should use a distributed lock - ReplLockManager, https://github.com/bakwc/PySyncObj#lock
Each node should try to acquire lock, and only a node that was able to acquire it should send notification
batkins61
@batkins61
Excellent, thanks!
batkins61
@batkins61
The example in lock.py looks like an implementation, not an example of ReplLockManager. Also, is there an article or other information on the semantics of this implementation?
Filipp Ozinov
@bakwc
I'm afraid there is no example except the one in README. Do you have some questions?
batkins61
@batkins61
It looks like a lock can only be acquired with the entire cluster is connected, right? We need to handle the case where a member may be down for an extended amount of time. Also, once acquired, does the lock remain with the holder until explicitly released or the holder looses connectivity to the rest of the cluster? In other words, is does the normal heartbeat process refresh the lock, or must it be renewed/reacquired explicitly?
Filipp Ozinov
@bakwc

It looks like a lock can only be acquired with the entire cluster is connected, right?

Only when the majority of the cluster is alive

Also, once acquired, does the lock remain with the holder until explicitly released or the holder looses connectivity to the rest of the cluster?

It will remain with the holder until explicitly released or the holder looses connectivity to the rest of the cluster

is does the normal heartbeat process refresh the lock

Exactly, it do it automatically every autoUnlockTime / 4 (autoUnlockTime is specified in ReplLockManager constructor)

quocbao
@baonq-me
Hello all
Are there any hook that can be triggered when leader election or leader-follower transistion happen ?
Filipp Ozinov
@bakwc
Currently there is no such hook, but it can be easily added (to syncobj.py, __onLeaderChanged). You cand add it and make pull-request.
quocbao
@baonq-me
Thanks, I'll

Another thing, on the example kvstorage_http.py

./kvstorage_http.py 8080 dump.bin.gz localhost:8081 localhost:8082 localhost:8083
ERROR:root:failed to load full dump
Traceback (most recent call last):
File "/home/quocbao/KhoaLuan/raft/pysyncobj/syncobj.py", line 1295, in loadDumpFile
data = self.
serializer.deserialize()
File "/home/quocbao/KhoaLuan/raft/pysyncobj/serializer.py", line 113, in deserialize
return pickle.load(g)
File "/home/quocbao/KhoaLuan/raft/pysyncobj/pickle.py", line 54, in load
return pickle.load(file)
EOFError

I create the empty dump file
I guess its inital content is the output of pickle.dumps({}) and then the file is compressed by gzip
Matt Hall
@mattall

Regarding locks: I have a replicated object (A) on a cluster of servers that clients will modify. The object has its own object, and that object is a graph (A.graph). Clients will modify attributes in the edges of the graph (A.graph.edge_color := 'blue'). Each edge will have a lock, so clients updating multiple edges will need to acquire all locks before making changes.

The edges of the graph are populated with a method after the object (A) is instantiated. If I want to protect my edges from race conditions at different servers I need assign a lock for each edge as the edge is created.

My question is: Should one lock manager oversee all of the locks, or should there be a lock manager for each lock?

quocbao
@baonq-me
As you said that "clients updating multiple edges will need to acquire all locks before making changes.", you only need one lock manager
quocbao
@baonq-me
This problem can be solved easily if you redirect all client request to a single server that is a leader in raft
Filipp Ozinov
@bakwc

Should one lock manager oversee all of the locks, or should there be a lock manager for each lock?

One lock manager is enough, it can handle multiple locks

Better solution will be to add a @repicated method to your object that modifies edges, eg:
@replicated
def changeEdgeColor(self, edgeID = 12345, edgeColor = 'blue'):
You don't need any lock to execute repicated methods, they always executed at the same order at all servers (but you need to handle case where edge with given id is already deleted - just skip editing for example, and all edgeID-s should be unique).
Matt Hall
@mattall

Thank you both! I am using replicated sync for all of my graph-changing methods.

My server has multiple replicated classes, so forwarding to the leader is not an option since the leader of the different classes might not be on the same server. Basically, I let the server who caught the request handle responding to it, and make sure that it uses replicated methods for changing anything across the cluster.

The thing is that with the graph object, I want to make updates to multiple edges atomic. For instance if multiple clients are trying to change edges on different paths, and those paths overlap with a common edge, then I only want one client to have access to all of those edges at any time.

@replicated_sync
def changeEdgeColor(self, edgeID = 12345, edgeColor = 'blue'):
    ...
    return "Success!"

@replicated_sync
def changeManyEdges(self, list_of_edges = [1, 2, 3] ):
     sort(list_of_edges)
     for edge in list_of_edges:
         self.__LockManager.tryAcquire(edge)

     for edge in list_of_edges:
         changeEdgeColor(self, edge, 'blue')

     for edge in list_of_edge:
         self.__LockManager.release(edge)

One question here about the lock manager. it seems like tryAcquire does not require me to create the lock myself. Is that true?

if existingLock is None or existingLock[0] == clientID:
            self.__locks[lockID] = (clientID, currentTime)
return True
Geert Audenaert
@FastGeert
Hi guys. I'm using pysyncobj in gevent application where i do monkey.patch_all() as the first line that gets executed. From time to time i get the following stack trace that messes up the application:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/dist-packages/gevent/threading.py", line 177, in run
    super(Thread, self).run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/syncobj.py", line 500, in _autoTickThread
    self._onTick(self.__conf.autoTickPeriod)
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/syncobj.py", line 577, in _onTick
    res = self.__doApplyCommand(entry[0])
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/syncobj.py", line 701, in __doApplyCommand
    return self._idToMethod[funcID](*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/syncobj.py", line 1389, in newFunc
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/batteries.py", line 419, in prolongate
    for lockID in self.__locks.keys():
RuntimeError: dictionary changed size during iteration
Could this be related to the use of gevent?
Filipp Ozinov
@bakwc
Hi, @FastGeert , thanks for report. It is bug in our library, created an issue, bakwc/PySyncObj#102
Filipp Ozinov
@bakwc
Fixed it, try version from github
Geert Audenaert
@FastGeert
Great, will do.
I'm really trilled by your project btw. I was wondering why consumers cannot be added after SyncObj is created?
MoayedHajiAli
@MoayedHajiAli
Hello everyone.
Hope everyone is fine.
I have a simple question about PySyncObj.
When sending too many RPS, the leader fails.
That happens in the provided benchmark as well.
It is weird for me though, as I expect more packets to be dropped when sending to many requests but not the leader to fail.
can anyone please explain why does this happen ?
Filipp Ozinov
@bakwc
What do you mean by leader fails?
MoayedHajiAli
@MoayedHajiAli
Hi.
Thanks for your fast reply and Sorry for the delay in the reply. I did not notice your reply.
I mean when sending a high RPS, in the RPS benchmark, some packets fail right?
and the FAIL_REASON that is reported is, in a high percentage, 5 (which indicates a LEADER_CHANGE)
However, it seems strange to me because I think that the leader should stay the same whatever the number of packets that are sent.
given a sufficient timeout, the leader should maintain its state and never change, unless the leader itself fail for any reason.