These are chat archives for ethersphere/orange-lounge

2nd
Oct 2017
Anish Mohammed
@anishmohammed_twitter
Oct 02 2017 03:24
@zelig any chance of ticket to devcon :)
Viktor Trón
@zelig
Oct 02 2017 06:36
@lmars if i use net.FileConn wrapped os.Pipe i get this:
TRACE[10-02|08:33:43] Dial error                               task="staticdial ddd2560407616c50 127.0.0.1:30303" err="file file+net |0: getsockopt: socket operation on non-socket"
Anton Evangelatov
@nonsense
Oct 02 2017 07:33
Hi everyone! I am excited to start to work with all of you!
lash
@nolash
Oct 02 2017 07:52
Hey @nonsense welcome :)
Viktor Trón
@zelig
Oct 02 2017 08:22
hey Anton welcome.
Lewis Marshall
@lmars
Oct 02 2017 09:06
@zelig what is the socket operation it is trying to do? It is indeed not a socket, it's a pipe. If we actually need a socket then on Linux we need to use socketpair(2), not sure about OSX though
lash
@nolash
Oct 02 2017 09:38
a
Viktor Trón
@zelig
Oct 02 2017 10:02
I just tried os.Pipe
Lewis Marshall
@lmars
Oct 02 2017 10:22
can you show me the code?
ah, ok perhaps os.Pipe is incompatible with net.FileConn then, I thought I'd used them together before but maybe not
so this is what I did on Linux to create two os.File using socketpair(2) which can then be wrapped in net.FileConn: https://github.com/flynn/flynn/blob/master/host/containerinit/init.go#L743-L749
it may well work on OSX, not sure
Viktor Trón
@zelig
Oct 02 2017 12:15
@nonsense you wanna hop on https://hangouts.google.com/hangouts/_/ethereum.org/swarm for a chat at 2.30pm?
Anton Evangelatov
@nonsense
Oct 02 2017 12:15
sure
Viktor Trón
@zelig
Oct 02 2017 12:15
:+1:
holisticode
@holisticode
Oct 02 2017 12:39
@nonsense welcome!
Viktor Trón
@zelig
Oct 02 2017 13:13
@/all we are on the call. simdemo?
holisticode
@holisticode
Oct 02 2017 13:13
here
Viktor Trón
@zelig
Oct 02 2017 13:18
lash
@nolash
Oct 02 2017 13:39
?!
lash
@nolash
Oct 02 2017 14:31
Question I tried to post/ask in hangout: hello?
I would like to know how many nodes have we actulaly been able to fire up simultanesouly by now?
My system starts struggling around 100
So what kind of hardware is needed for thousands of nodes?!
Anton Evangelatov
@nonsense
Oct 02 2017 14:32
I haven’t given it a tried with that many nodes yet. Will see and comment back when I try.
lash
@nolash
Oct 02 2017 14:32
Seems the even the chat msgs don't always make it through in hangouts. Frustrating!
holisticode
@holisticode
Oct 02 2017 14:35
@nolash by "fire up simultaneously" you mean through the frontend? Or e.g. via discover_test in the backend only?
Anton Evangelatov
@nonsense
Oct 02 2017 14:35
I did see this question from you. Just didn’t have a clue about the answer :) sorry
lash
@nolash
Oct 02 2017 14:35
backend first and foremost @holisticode. thanks for the presentation, byt the way :)
holisticode
@holisticode
Oct 02 2017 14:35
thank you for attending :)
lash
@nolash
Oct 02 2017 14:35
i have an eerie feeling that the implementation is a bit on the heavy side
maybe we should have a pass at making the code leaner. at least see what resources are consumed per node, and maybe some more proactive memory management - if golang allows it
holisticode
@holisticode
Oct 02 2017 14:36
I have tried 100 nodes no problem, and I believe I once ran discovery_test with 500 (that was a while back)
with 500 it take quite a long time
lash
@nolash
Oct 02 2017 14:36
yeah and then imagine activity afterwards between those nodes...
holisticode
@holisticode
Oct 02 2017 14:37
Maybe we should also take a different stance
like running these tests in the cluster and distribute load over several machines?
lash
@nolash
Oct 02 2017 14:37
yeah, but I think there is something to be said for opmimization of code to
*too
holisticode
@holisticode
Oct 02 2017 14:38
No doubt about that
lash
@nolash
Oct 02 2017 14:38
at least gauging the resource use. there seems to be little focus on that. too little, in my opinion
holisticode
@holisticode
Oct 02 2017 14:38
Just pointing to the fact that running all on one machine is great for developing
lash
@nolash
Oct 02 2017 14:38
yeah for sure
holisticode
@holisticode
Oct 02 2017 14:38
but as soon as we want any feedback on actual performance etc we should probably consider different "layout" as well
Lewis Marshall
@lmars
Oct 02 2017 14:39
I think having 100 in-memory nodes all running on a laptop and communicating over encrypted p2p connections is quite good, I probably wouldn't expect much better
holisticode
@holisticode
Oct 02 2017 14:39
agreed
Viktor Trón
@zelig
Oct 02 2017 14:39
by the way
Lewis Marshall
@lmars
Oct 02 2017 14:39
the plan was to have a "cluster adapter" so you could run bigger simulations in AWS / Azure / GCE for example
Viktor Trón
@zelig
Oct 02 2017 14:39
I managed to fix the net pipe
pss tests work
but
lash
@nolash
Oct 02 2017 14:40
@zelig that's cool, will you push it please
oh
Viktor Trón
@zelig
Oct 02 2017 14:40
I'm struggling to figure out
how to set ulimit for a user
could someone figure out?
putting
* - nofile 10000
in /etc/security/limits.conf
didnt work on ubuntu
it does for @nonsense though...
lash
@nolash
Oct 02 2017 14:44
dunno but: Note: Using /etc/security/limits.conf is deprecated. Please use /etc/systemd/system.conf, /etc/systemd/user.conf, and /etc/systemd/<systemd_unit>/override.conf to control the resources. See systemd-system.conf man page for details.
from arch
holisticode
@holisticode
Oct 02 2017 14:44
like
sudo sysctl -w fs.inotify.max_user_watches=100000?
@zelig when do you want to talk about the diagrams for the paper?
Viktor Trón
@zelig
Oct 02 2017 14:47
in about 40-45 mins i back
holisticode
@holisticode
Oct 02 2017 14:47
cool
lash
@nolash
Oct 02 2017 14:57
@zelig I had to manually ulimit -n <num> after setting new limit in limits.conf
Anton Evangelatov
@nonsense
Oct 02 2017 15:14
it did work for me, but inside a docker container, i wouldn’t be surprised if its different on your machine
also on OS X I didn’t need root permissions to change it on the host
Viktor Trón
@zelig
Oct 02 2017 15:51
so why does it not work for me??
Anton Evangelatov
@nonsense
Oct 02 2017 15:52
do you just change your user after editing the file with su tron or you start a new terminal? could something be sourced that resets the ulimit?
Viktor Trón
@zelig
Oct 02 2017 15:53
oh wait
so you say security/limits.conf is deprecated
Anton Evangelatov
@nonsense
Oct 02 2017 15:56
Viktor Trón
@zelig
Oct 02 2017 16:01
that is the problem
already the first line
root: ulimit -n
1024
Anton Evangelatov
@nonsense
Oct 02 2017 16:05
maybe this PAM module is not enabled… here they suggest adding session required pam_limits.so to /etc/pam.d/common-session - https://askubuntu.com/questions/753475/cant-set-max-limit-of-opened-files
i guess if this PAM module is not enabled, editing /etc/security/limits.conf has no effect
thats just a guess though
same suggestion - https://ubuntuforums.org/showthread.php?t=1437625 - but you’d need to reboot again
holisticode
@holisticode
Oct 02 2017 16:47
regarding the circular diagram displaying connection relationships:
this type of diagram is called "Chord" diagram
from the docs:
Computes the chord layout for the specified square matrix of size n×n, where the matrix represents the directed flow amongst a network (a complete digraph) of n nodes. The given matrix must be an array of length n, where each element matrix[i] is an array of n numbers, where each matrix[i][j] represents the flow from the ith node in the network to the jth node.
this is the relevant code how to build the matric:
matrix[i][j] = connsById[id].connCount;
i is the source, j the target of a connection
a connection has an id, which has the format id: event.conn.one + "-" + event.conn.other,
Viktor Trón
@zelig
Oct 02 2017 16:52
@holisticode you wanna hop on the swarm call for a sec?
holisticode
@holisticode
Oct 02 2017 16:53
so if the connection events correctly represent who is oneand who is other, then the matrix is representing cumulated directed connection events from one node to the other
sure
holisticode
@holisticode
Oct 02 2017 17:26
@zelig here the function (in javascript on the client side) which calculates the distance between two nodes of a connection:
el.distance = (10 - DefaultPof(nodesById[el.source].overlay, nodesById[el.target].overlay,0)) * 10;
I have ported DefaultPof from Go to javascript like so:
function DefaultPof(one, other, pos) {
  const MAX = 256
  let po = proximityOrder(ToBytes(one), ToBytes(other), pos);
  if (po >= MAX) {
        po = MAX;
  }
  return po
}
and proximityOrder:
function proximityOrder(one, other, pos) {
    for (let i=pos / 8; i< one.length; i++) {
        if (one[i] == other[i]) {
            continue
        }
        let oxo = one[i] ^ other[i];
        let start = 0;
        if (i == pos/8) {
            start = pos % 8;
        }
        for (let j=start; j<8; j++) {
            if (oxo>>(7-j)&0x01 != 0) {
                return i*8 + j;
            }
        }
    }
    return one.length * 8
}
this might need to be reviewed, as you commented that it might not be fully correct
lash
@nolash
Oct 02 2017 17:45
@zelig did you push the changes?
holisticode
@holisticode
Oct 02 2017 18:17
question: is it possible that a msgbe sent from one to other when a connection hasn't been established yet between one and other? Maybe discovery?
@lmars @nolash ?
lash
@nolash
Oct 02 2017 18:17
handshake?
holisticode
@holisticode
Oct 02 2017 18:18
yeah I was thinking smth like that
ok so it seems fine
thanks @nolash
lash
@nolash
Oct 02 2017 18:19
:)
Lewis Marshall
@lmars
Oct 02 2017 18:19
@holisticode the connection event will always happen "before" the message is sent, but I don't think there is a guarantee that the events are delivered in order
holisticode
@holisticode
Oct 02 2017 18:20
ah ok
Lewis Marshall
@lmars
Oct 02 2017 18:20
are you still blocked on anything for your PR?
holisticode
@holisticode
Oct 02 2017 18:20
not really
Lewis Marshall
@lmars
Oct 02 2017 18:20
@nolash did you manage to push that logging fix for the ExecAdapter?
@holisticode ok, so do you want me to do another review pass?
lash
@nolash
Oct 02 2017 18:21
Yeah its in the pss-networktest @lmars
holisticode
@holisticode
Oct 02 2017 18:21
I will have to remove the unnecessary go routine as you pointed out
lash
@nolash
Oct 02 2017 18:21
I'm fixing the swarm.go file now. I thin kit's the third time I'm reinstating the same changes in there... :/
holisticode
@holisticode
Oct 02 2017 18:21

```
if connCount == (nodeCount-1)*2 {
wg.Done()
return

```

Lewis Marshall
@lmars
Oct 02 2017 18:21
@nolash can you please push that fix to this PR: ethereum/go-ethereum#15198
(that is where we introduced the bug, so we shouldn't merge it without the fix)
lash
@nolash
Oct 02 2017 18:22
as-is?
ok
holisticode
@holisticode
Oct 02 2017 18:22
and have an informed opinion on the above code, if it is safe to wait for that many connCount before we can consider that the mocker completed the connection phase...
As soon as I have amended the go routine thing I'll ping you and if needed have a new review @lmars
Lewis Marshall
@lmars
Oct 02 2017 18:25
@holisticode ok, perhaps remove the goroutine and ping me then I'll review it
holisticode
@holisticode
Oct 02 2017 18:25
cool