Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 09:35
    bakwc commented #133
  • 09:23
    CyberDem0n commented #133
  • 09:16

    bakwc on master

    Bugfix: use == for comparison w… Merge pull request #131 from Cy… (compare)

  • 09:16
    bakwc closed #131
  • 09:14

    bakwc on master

    Use node.ip instead of host for… Merge pull request #132 from Cy… (compare)

  • 09:14
    bakwc closed #132
  • 09:10
    bakwc commented #133
  • 08:06
    CyberDem0n opened #133
  • 08:01
    coveralls commented #132
  • 07:34
    coveralls commented #131
  • 07:01
    CyberDem0n opened #132
  • 07:00
    CyberDem0n opened #131
  • Oct 26 15:13

    bakwc on master

    Refactor syncobj_admin Extract… Address review feedback Introduce UtilityException and … and 1 more (compare)

  • Oct 26 15:13
    bakwc closed #129
  • Oct 26 14:56
    CyberDem0n synchronize #129
  • Oct 26 13:58
    coveralls commented #129
  • Oct 26 13:19
    CyberDem0n synchronize #129
  • Oct 26 12:06
    bakwc opened #130
  • Oct 26 11:39

    bakwc on master

    Introduce utility message callb… Merge pull request #128 from Cy… (compare)

  • Oct 26 11:39
    bakwc closed #128
marlonrodriguezpinto
@marlonrodriguezpinto
Hi again, hopefully you can help me out again. I want to count the number of message that each node sends and receives if for example the counter value has changed. The goal is to check if the theoretical number of messages matches the implemented number of message which are needed to achieve consensus with the Raft algorithm. Thank you for any hint how to measure that.
Filipp Ozinov
@bakwc
Hi, you should look at syncobj.py:1151 __send and syncobj.py:703 _onMessageReceived - just add counter and increment it in this methods. Also it is quite useful metric, it would be great if you add it to node statistics (getStatus method, syncobj.py:621) and make a pull request with this improvement.
Gabbi Fisher
@gabbifish
Hi there! I have two python scripts running on my machine's localhost, but on different ports. I'm trying to propagate a python dict between the two scripts (set a key-value pair on one instance and read that new key-value pair in another instance). How do I do this?
I use the ReplDict() battery and its get and set methods
Nvm, got it!
Matt Hall
@mattall

Hey all, just looking for some pointers getting started with syncobj. First, If I run the example 'kvstorage.py' how can get multiple servers running? I ran 'python kvstorage.py 127.0.0.1:9999 127.0.0.2:9999 127.0.0.3:9999'. Do I have to run a similar command in another terminal with the self address and partner addresses change? When I tried doing that I got an error "OSError: [Errno 49] Can't assign requested address"

Secondly, after running the first command (python kvstorage.py 127.0.0.1:9999 127.0.0.2:9999 127.0.0.3:9999') I tried to do ">>> set name 'matt'" but "get name" returned "None".

Thanks for any pointers. I'm really excited to start using SyncObj for my project if I can just get these little things figured out.

Filipp Ozinov
@bakwc
Hi, first - you should use different ports, not ip. You have only one local ip, 127.0.0.1, there is no ip 127.0.0.2 or 127.0.0.3. And you should change the first argument (self address), cause you cant have the same self address for two apps. So in terminal1 you should run: kvstorage.py 127.0.0.1:9999 127.0.0.1:9998 127.0.0.1:9997, in terminal2 kvstorage.py 127.0.0.1:9998 127.0.0.1:9997 127.0.0.1:9999 and in terminal3 kvstorage.py 127.0.0.1:9997 127.0.0.1:9999 127.0.0.1:9998
Second - for 3-node cluster you need at least two nodes up. In you case no connection was established with other nodes.
Matt Hall
@mattall
Thanks, that helps. I'm trying to use this for my client server app. How can I define a handle() method in kvstorage to use the tcp server that SyncObj creates? I tried modifying kvstorage-httpserver to work for TCP connections but get an "Address already in use" error. Instead of httpServer = HTTPServer(('', httpPort), KVRequestHandler) I'm doing TCPServer = socketserver.TCPServer((selfHost,int(selfPort)), KVRRequestHandler)If I can get a generic version of this to work I'll gladly make a pull request to add it as en example, but there's also a chance that I'm just approaching this the completely wrong way.
Filipp Ozinov
@bakwc
You can't have two tcp servers on the same port. You need to use another port for TCPServer, (selfHost,int(selfPort) is already used for SyncObj.
Matt Hall
@mattall
The TcpServer for SyncObj doesn't have a handler. Can I write my own handler for it, or would it be easier to just create a new server on a new port to let clients interact with the cluster from there?
Filipp Ozinov
@bakwc
It would be easier to create a new server on a new port for clients.
AmithGopal
@amith1893
How do I stop just one of the servers out of all the servers running ?
Matt Hall
@mattall
kill -9 <procid>
AmithGopal
@amith1893
Thanks. @mattall I want to kill the leader of the process cluster
*server
How do I find which server is the leader ?
Matt Hall
@mattall
you can call getStatus, a SyncObj method. This will give you info about the cluster. You can also call _getLeader
AmithGopal
@amith1893
Thanks @mattall
AmithGopal
@amith1893
how to make any function call on the Class Derived from SyncObject blocking ?
Filipp Ozinov
@bakwc
functions marked by @replicated decorator blocking by default. Or you can pass argument sync=True, and no callback argument.
batkins61
@batkins61
I have an application where multiple nodes receive an event, and one of them should then send out a notification. My current plan is to keep what notifications have already been sent in a ReplDict and use leadership to determine which node actually sends the notification. However, I see comments in the code that warn that there could be multiple leaders. I'm trying to understand how this can come about in this implementation and how to defend against it. Any other advice is most welcome as well.
Filipp Ozinov
@bakwc
This is possible due to raft protocol limitations. There is no guarantee that there is only one leader at any time moment. For example, when your cluster is splitted into several parts due to some network issues, a part with the current leader will still think that leader has not changed, and another part can elect another leader.
You should use a distributed lock - ReplLockManager, https://github.com/bakwc/PySyncObj#lock
Each node should try to acquire lock, and only a node that was able to acquire it should send notification
batkins61
@batkins61
Excellent, thanks!
batkins61
@batkins61
The example in lock.py looks like an implementation, not an example of ReplLockManager. Also, is there an article or other information on the semantics of this implementation?
Filipp Ozinov
@bakwc
I'm afraid there is no example except the one in README. Do you have some questions?
batkins61
@batkins61
It looks like a lock can only be acquired with the entire cluster is connected, right? We need to handle the case where a member may be down for an extended amount of time. Also, once acquired, does the lock remain with the holder until explicitly released or the holder looses connectivity to the rest of the cluster? In other words, is does the normal heartbeat process refresh the lock, or must it be renewed/reacquired explicitly?
Filipp Ozinov
@bakwc

It looks like a lock can only be acquired with the entire cluster is connected, right?

Only when the majority of the cluster is alive

Also, once acquired, does the lock remain with the holder until explicitly released or the holder looses connectivity to the rest of the cluster?

It will remain with the holder until explicitly released or the holder looses connectivity to the rest of the cluster

is does the normal heartbeat process refresh the lock

Exactly, it do it automatically every autoUnlockTime / 4 (autoUnlockTime is specified in ReplLockManager constructor)

quocbao
@baonq-me
Hello all
Are there any hook that can be triggered when leader election or leader-follower transistion happen ?
Filipp Ozinov
@bakwc
Currently there is no such hook, but it can be easily added (to syncobj.py, __onLeaderChanged). You cand add it and make pull-request.
quocbao
@baonq-me
Thanks, I'll

Another thing, on the example kvstorage_http.py

./kvstorage_http.py 8080 dump.bin.gz localhost:8081 localhost:8082 localhost:8083
ERROR:root:failed to load full dump
Traceback (most recent call last):
File "/home/quocbao/KhoaLuan/raft/pysyncobj/syncobj.py", line 1295, in loadDumpFile
data = self.
serializer.deserialize()
File "/home/quocbao/KhoaLuan/raft/pysyncobj/serializer.py", line 113, in deserialize
return pickle.load(g)
File "/home/quocbao/KhoaLuan/raft/pysyncobj/pickle.py", line 54, in load
return pickle.load(file)
EOFError

I create the empty dump file
I guess its inital content is the output of pickle.dumps({}) and then the file is compressed by gzip
Matt Hall
@mattall

Regarding locks: I have a replicated object (A) on a cluster of servers that clients will modify. The object has its own object, and that object is a graph (A.graph). Clients will modify attributes in the edges of the graph (A.graph.edge_color := 'blue'). Each edge will have a lock, so clients updating multiple edges will need to acquire all locks before making changes.

The edges of the graph are populated with a method after the object (A) is instantiated. If I want to protect my edges from race conditions at different servers I need assign a lock for each edge as the edge is created.

My question is: Should one lock manager oversee all of the locks, or should there be a lock manager for each lock?

quocbao
@baonq-me
As you said that "clients updating multiple edges will need to acquire all locks before making changes.", you only need one lock manager
quocbao
@baonq-me
This problem can be solved easily if you redirect all client request to a single server that is a leader in raft
Filipp Ozinov
@bakwc

Should one lock manager oversee all of the locks, or should there be a lock manager for each lock?

One lock manager is enough, it can handle multiple locks

Better solution will be to add a @repicated method to your object that modifies edges, eg:
@replicated
def changeEdgeColor(self, edgeID = 12345, edgeColor = 'blue'):
You don't need any lock to execute repicated methods, they always executed at the same order at all servers (but you need to handle case where edge with given id is already deleted - just skip editing for example, and all edgeID-s should be unique).
Matt Hall
@mattall

Thank you both! I am using replicated sync for all of my graph-changing methods.

My server has multiple replicated classes, so forwarding to the leader is not an option since the leader of the different classes might not be on the same server. Basically, I let the server who caught the request handle responding to it, and make sure that it uses replicated methods for changing anything across the cluster.

The thing is that with the graph object, I want to make updates to multiple edges atomic. For instance if multiple clients are trying to change edges on different paths, and those paths overlap with a common edge, then I only want one client to have access to all of those edges at any time.

@replicated_sync
def changeEdgeColor(self, edgeID = 12345, edgeColor = 'blue'):
    ...
    return "Success!"

@replicated_sync
def changeManyEdges(self, list_of_edges = [1, 2, 3] ):
     sort(list_of_edges)
     for edge in list_of_edges:
         self.__LockManager.tryAcquire(edge)

     for edge in list_of_edges:
         changeEdgeColor(self, edge, 'blue')

     for edge in list_of_edge:
         self.__LockManager.release(edge)

One question here about the lock manager. it seems like tryAcquire does not require me to create the lock myself. Is that true?

if existingLock is None or existingLock[0] == clientID:
            self.__locks[lockID] = (clientID, currentTime)
return True
Geert Audenaert
@FastGeert
Hi guys. I'm using pysyncobj in gevent application where i do monkey.patch_all() as the first line that gets executed. From time to time i get the following stack trace that messes up the application:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/dist-packages/gevent/threading.py", line 177, in run
    super(Thread, self).run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/syncobj.py", line 500, in _autoTickThread
    self._onTick(self.__conf.autoTickPeriod)
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/syncobj.py", line 577, in _onTick
    res = self.__doApplyCommand(entry[0])
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/syncobj.py", line 701, in __doApplyCommand
    return self._idToMethod[funcID](*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/syncobj.py", line 1389, in newFunc
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pysyncobj/batteries.py", line 419, in prolongate
    for lockID in self.__locks.keys():
RuntimeError: dictionary changed size during iteration
Could this be related to the use of gevent?
Filipp Ozinov
@bakwc
Hi, @FastGeert , thanks for report. It is bug in our library, created an issue, bakwc/PySyncObj#102
Filipp Ozinov
@bakwc
Fixed it, try version from github
Geert Audenaert
@FastGeert
Great, will do.