These are chat archives for atomix/atomix

3rd
Mar 2017
Jordan Halterman
@kuujo
Mar 03 2017 00:03
okay, I got a fix for it @jhall11 just need to write some more tests to make sure I’m not doing something stupid
Jon Hall
@jhall11
Mar 03 2017 00:03
great!
Jordan Halterman
@kuujo
Mar 03 2017 00:06
I think this is going to work though
Jon Hall
@jhall11
Mar 03 2017 00:14
So I think one of the possible causes of the No handler issue is that there was no close listener for the CopycatTransportClient in ONOS. I’m testing now to see if that makes these dissapear
Jordan Halterman
@kuujo
Mar 03 2017 00:19
I think this looks pretty good!
going to open a PR and then think about it a bit
Jon Hall
@jhall11
Mar 03 2017 00:26
ok
Jordan Halterman
@kuujo
Mar 03 2017 00:33
I’m still not totally sure about this. I think this condition may actually need to be event.request.previousIndex <= eventIndex but I need to ponder for a bit
but it’s in atomix/copycat#293
I think that type of scenario is possible
Jon Hall
@jhall11
Mar 03 2017 00:37
hmm, looks like the adding a close listener fixed the No handler issue. I’ll submit a fix
err never mind, I’m still seeing them, just not as often...
Jordan Halterman
@kuujo
Mar 03 2017 00:51
hmm I think I may know what the problem is
@jhall11 So, I think what the problem may be is when the client side of a CopycatTransportConnection is closed, nothing is notifying the server side. When this is done through TCP using Copycat’s NettyTransport, the client will close a connection and the server side of the connection’s closeListener will be called (when the TCP connection is closed), causing the server to unregister the connection. The reason we probably didn’t see this much before is because prior to atomix/copycat#276 the servers were largely tracking connections through the Raft log, so the servers could pretty quickly resolve which node the client was connected to. WIthout a client-side close() calling closeListener on the server side, those servers are assuming the connection is still open. So, they’re sending PublishRequest assuming the client side is still listening, while the client-side has been close()ed and thus removed the handlers for that connection. So, what needs to be done is a close event needs to be sent from the client to the server or vice versa to ensure listeners on the other side of the connection are called.
Jordan Halterman
@kuujo
Mar 03 2017 00:57
I think that’s probably it
this just needs to send a notification to the other side that the connection has been closed
Jon Hall
@jhall11
Mar 03 2017 01:00
hmm, that makes sense.
Jordan Halterman
@kuujo
Mar 03 2017 01:01
I can fix that if you want
Jon Hall
@jhall11
Mar 03 2017 01:01
What is strange to me, is that I added a close listener that removes the connection for the set of connections. The remove call is returning true, but then the set still contains the connection
Jordan Halterman
@kuujo
Mar 03 2017 01:01
whoah weird
Jon Hall
@jhall11
Mar 03 2017 01:03
anyway, if you wouldn’t mind adding the close notification, my repo is pretty dirty right now
Jordan Halterman
@kuujo
Mar 03 2017 01:03
yeah
Jon Hall
@jhall11
Mar 03 2017 01:04
The more I look at this the more confused I’m getting. It looks like the close is getting called multiple times per connection
Jordan Halterman
@kuujo
Mar 03 2017 01:05
hmm… can you see where it’s called from?
k I’m going to try to hack out a fix for that and will also fix the consistent map UpdateAndGet command
then we hopefully will have all three problems fixed
Jordan Halterman
@kuujo
Mar 03 2017 01:41
I pushed the UpdateAndGet change
Jon Hall
@jhall11
Mar 03 2017 01:42
cool
So I think I just needed to split the logs up. I was getting confused as to what was on the client and what was on the server
It is looking like the closeListeners are getting executed later than I though they would be, but they are getting called and the connections are removed
Jordan Halterman
@kuujo
Mar 03 2017 01:44
:+1:
damn no tests for this transport
I think I can probably steal from the NettyMessagingManager tests to write a test
Jordan Halterman
@kuujo
Mar 03 2017 01:50
otherwise, I haven’t found any tests that hit create those connections… yet
Jordan Halterman
@kuujo
Mar 03 2017 02:03
hmm… this is a little more complicated by the structure of the server side since it listens on only one address but should still be good
just going to have to add another message type, just may take some time for me to figure out how to test it right :-)
Jordan Halterman
@kuujo
Mar 03 2017 02:13
@jhall11 I don’t know if this works yet, but it’s close: kuujo/onos@78ab161
It’s the easiest way to go about doing this. No need to invest too much in that transport right now. But I don’t know if it works yet until I can see it in some tests. Whether it works depends on how the Serializer is configured.
Jon Hall
@jhall11
Mar 03 2017 02:23
Sweet, we might even be able to get this into the release. I think the last(hopefully) rc will be noon tomorrow
Jordan Halterman
@kuujo
Mar 03 2017 02:28
:clap:
Well, I actually set the sequencer state to exactly what it was in the ONOS logs with the same responses and events, and it fixed the problem. Hopefully that's the end of it!
Jordan Halterman
@kuujo
Mar 03 2017 02:34
The tests are just a dumbed down version of that state plus a more convoluted version of that scenario. They all pass
But it's always hard to tell what a faulty distributed system will come up with next :-)
I suppose that's what model checking is for
I just don't have time for that
Haha
Jon Hall
@jhall11
Mar 03 2017 02:35
:)
Jon Hall
@jhall11
Mar 03 2017 02:49
so I uploaded the change listener plus some random bits
Jordan Halterman
@kuujo
Mar 03 2017 03:23
sweet
Jordan Halterman
@kuujo
Mar 03 2017 06:50
the transport actually needed a bit more work than I thought, but this should help a lot if I can get it right
Jordan Halterman
@kuujo
Mar 03 2017 08:18
I’ll push the code for review in just a bit
Jordan Halterman
@kuujo
Mar 03 2017 10:05

@jhall11 I ended up having to rewrite a large portion of the CopycatTransport. I think that change probably makes some of your changes obsolete if it can get merged...

https://gerrit.onosproject.org/#/c/12998/

I added tests for the Copycat transport as well. The MessagingService should be mocked, but I’m not familiar with the mocking framework used in ONOS and don’t want to try to figure it out in one night (though I probably could have). The tests I wrote are the closest I can get to verifying that the changes work since I don’t really have a good dev environment set up for ONOS yet.

Jordan Halterman
@kuujo
Mar 03 2017 10:20

The reason I had to rewrite a large portion of the transport is because connection events didn't really exist either. The server was more or less just creating connections lazily when a message was received. But this could perhaps risk recreating connections that are already closed. I'd prefer to see a dedicated message for opening/closing a connection.

So, I actually ended up removing the connect/close requests I was using and instead extended the protocol that's used inside the Copycat transport. I just added a 1-byte flag to requests to handle OPEN and CLOSE events and normal messages. I also made the code a little more aesthetically pleasing IMO ;-)

There are some inefficiencies in the transport, but they're fixed in the new Protocol that's in the Copycat 2.0 branch so I'm not worrying about them now.

Jon Hall
@jhall11
Mar 03 2017 17:47
ok, I’ll take a look now. It looks like my change was merged already so you will probably need to rebase on top of that
Jon Hall
@jhall11
Mar 03 2017 18:20
So it looks like the tests failed at around 80 iterations, but they were started with the == and not the <=. I’ll look at the logs and see if it looks like the same bug
Jordan Halterman
@kuujo
Mar 03 2017 18:44
I definitely did determine that there were some scenarios where == was not good enough to fix the consistency issue. It just took the client switching multiple servers, which again is just made more likely by the transport issue. I'm curious to see what that failure is from though
Jon Hall
@jhall11
Mar 03 2017 18:47
in case you wanted to look at them, I probably wont get a chance until after lunch
Jordan Halterman
@kuujo
Mar 03 2017 18:48
It also, BTW, appeared that at least some of the queries in earlier runs failed without the client sequencer actually stopping. The clients were able to continue to progress further down in the logs, but individual queries failed since they weren't able to sync their state with the cluster before the queries timed out. Those failures seemed to just be from the transport issue.
Once the sequencing gets out of sync from the transport issue, the client has to submit a keep-alive to reset the event index and then the server has to resend events, all before the query timed out. That seemed to cause some timeouts for queries that would have eventually succeeded. There's probably some improvement to be made to force the client to speed up that process.
But that's all conjecture. I'll look really quick
Jordan Halterman
@kuujo
Mar 03 2017 19:01
@jhall11 this one’s easy
2017-03-03 05:54:59,722 | DEBUG | artition-1-state | ServerSessionContext             | 90 - io.atomix.all - 1.0.3.SNAPSHOT | 174435 - Sending PublishRequest[session=174435, eventIndex=176325, previousIndex=176316, events=[Event[event=changeEvents, message=InstanceEvent[resource=161, message=[MapEvent{name=, type=UPDATE, key=8e:02:01:00:07:6f:72:67:2e:6f:6e:6f:73:70:72:6f:6a:65:63:74:2e:6f:70:74:69:63:61:6c:2d:6d:6f:64:65:ec, newValue=Versioned{value=byte[]{length=7906, hash=924500721}, version=58, creationTime=2017-03-03T05:51:28.227Z}, oldValue=Versioned{value=byte[]{length=7906, hash=924500720}, version=30, creationTime=2017-03-03T05:51:28.227Z}}]]]]]
2017-03-03 05:54:59,722 | DEBUG | artition-1-state | NettyMessagingManager            | 129 - org.onosproject.onos-core-dist - 1.10.0.SNAPSHOT | No handler for message type onos-copycat-1-1587099225769007480
2017-03-03 05:54:59,722 | DEBUG | artition-1-state | ServerSessionContext             | 90 - io.atomix.all - 1.0.3.SNAPSHOT | 174435 - Sending PublishRequest[session=174435, eventIndex=176327, previousIndex=176325, events=[Event[event=changeEvents, message=InstanceEvent[resource=161, message=[MapEvent{name=, type=UPDATE, key=8e:02:01:00:09:6f:72:67:2e:6f:6e:6f:73:70:72:6f:6a:65:63:74:2e:6f:70:65:6e:66:6c:6f:77:2d:62:61:73:e5, newValue=Versioned{value=byte[]{length=7930, hash=179985645}, version=59, creationTime=2017-03-03T05:51:28.246Z}, oldValue=Versioned{value=byte[]{length=7930, hash=179985644}, version=31, creationTime=2017-03-03T05:51:28.246Z}}]]]]]
2017-03-03 05:54:59,723 | DEBUG | artition-1-state | NettyMessagingManager            | 129 - org.onosproject.onos-core-dist - 1.10.0.SNAPSHOT | No handler for message type onos-copycat-1-1587099225769007480
2017-03-03 05:54:59,723 | DEBUG | artition-1-state | ServerSessionContext             | 90 - io.atomix.all - 1.0.3.SNAPSHOT | 174435 - Sending PublishRequest[session=174435, eventIndex=176329, previousIndex=176327, events=[Event[event=changeEvents, message=InstanceEvent[resource=161, message=[MapEvent{name=, type=UPDATE, key=8e:02:01:00:19:6f:72:67:2e:6f:6e:6f:73:70:72:6f:6a:65:63:74:2e:64:72:69:76:65:72:f3, newValue=Versioned{value=byte[]{length=7886, hash=1513221058}, version=60, creationTime=2017-03-03T05:51:28.341Z}, oldValue=Versioned{value=byte[]{length=7886, hash=1513221057}, version=29, creationTime=2017-03-03T05:51:28.341Z}}]]]]]
2017-03-03 05:54:59,723 | DEBUG | artition-1-state | NettyMessagingManager            | 129 - org.onosproject.onos-core-dist - 1.10.0.SNAPSHOT | No handler for message type onos-copycat-1-1587099225769007480
The client just doesn’t even realize those PublishRequests are lost and doesn’t switch servers in time for the resolution process to happen, so
org.onosproject.store.service.StorageException$Timeout
    at org.onosproject.store.primitives.DefaultLeaderElector.complete(DefaultLeaderElector.java:115)
    at org.onosproject.store.primitives.DefaultLeaderElector.getLeaderships(DefaultLeaderElector.java:80)
    at org.onosproject.store.cluster.impl.DistributedLeadershipStore.getLeaderships(DistributedLeadershipStore.java:170)
    at org.onosproject.cluster.impl.LeadershipManager.getLeaderBoard(LeadershipManager.java:98)
    at org.onosproject.cli.net.LeaderCommand.execute(LeaderCommand.java:140)
    at org.onosproject.cli.AbstractShellCommand.doExecute(AbstractShellCommand.java:150)

ditto all the other ones:

2017-03-03 05:55:07,234 | DEBUG | artition-1-state | ServerSessionContext             | 90 - io.atomix.all - 1.0.3.SNAPSHOT | 174435 - Sending PublishRequest[session=174435, eventIndex=176325, previousIndex=176316, events=[Event[event=changeEvents, message=InstanceEvent[resource=161, message=[MapEvent{name=, type=UPDATE, key=8e:02:01:00:07:6f:72:67:2e:6f:6e:6f:73:70:72:6f:6a:65:63:74:2e:6f:70:74:69:63:61:6c:2d:6d:6f:64:65:ec, newValue=Versioned{value=byte[]{length=7906, hash=924500721}, version=58, creationTime=2017-03-03T05:51:28.227Z}, oldValue=Versioned{value=byte[]{length=7906, hash=924500720}, version=30, creationTime=2017-03-03T05:51:28.227Z}}]]]]]
2017-03-03 05:55:07,234 | DEBUG | artition-1-state | NettyMessagingManager            | 129 - org.onosproject.onos-core-dist - 1.10.0.SNAPSHOT | No handler for message type onos-copycat-1-1587099225769007480
2017-03-03 05:55:07,234 | DEBUG | artition-1-state | ServerSessionContext             | 90 - io.atomix.all - 1.0.3.SNAPSHOT | 174435 - Sending PublishRequest[session=174435, eventIndex=176327, previousIndex=176325, events=[Event[event=changeEvents, message=InstanceEvent[resource=161, message=[MapEvent{name=, type=UPDATE, key=8e:02:01:00:09:6f:72:67:2e:6f:6e:6f:73:70:72:6f:6a:65:63:74:2e:6f:70:65:6e:66:6c:6f:77:2d:62:61:73:e5, newValue=Versioned{value=byte[]{length=7930, hash=179985645}, version=59, creationTime=2017-03-03T05:51:28.246Z}, oldValue=Versioned{value=byte[]{length=7930, hash=179985644}, version=31, creationTime=2017-03-03T05:51:28.246Z}}]]]]]
2017-03-03 05:55:07,234 | DEBUG | artition-1-state | NettyMessagingManager            | 129 - org.onosproject.onos-core-dist - 1.10.0.SNAPSHOT | No handler for message type onos-copycat-1-1587099225769007480
2017-03-03 05:55:07,234 | DEBUG | artition-1-state | ServerSessionContext             | 90 - io.atomix.all - 1.0.3.SNAPSHOT | 174435 - Sending PublishRequest[session=174435, eventIndex=176329, previousIndex=176327, events=[Event[event=changeEvents, message=InstanceEvent[resource=161, message=[MapEvent{name=, type=UPDATE, key=8e:02:01:00:19:6f:72:67:2e:6f:6e:6f:73:70:72:6f:6a:65:63:74:2e:64:72:69:76:65:72:f3, newValue=Versioned{value=byte[]{length=7886, hash=1513221058}, version=60, creationTime=2017-03-03T05:51:28.341Z}, oldValue=Versioned{value=byte[]{length=7886, hash=1513221057}, version=29, creationTime=2017-03-03T05:51:28.341Z}}]]]]]
2017-03-03 05:55:07,235 | DEBUG | artition-1-state | NettyMessagingManager            | 129 - org.onosproject.onos-core-dist - 1.10.0.SNAPSHOT | No handler for message type onos-copycat-1-1587099225769007480

and then

org.onosproject.store.service.ConsistentMapException$Timeout: onos-meter-store
    at org.onosproject.store.primitives.DefaultConsistentMap.complete(DefaultConsistentMap.java:228)[125:org.onosproject.onos-api:1.10.0.SNAPSHOT]
    at org.onosproject.store.primitives.DefaultConsistentMap.values(DefaultConsistentMap.java:143)[125:org.onosproject.onos-api:1.10.0.SNAPSHOT]
    at org.onosproject.store.primitives.ConsistentMapBackedJavaMap.values(ConsistentMapBackedJavaMap.java:141)[125:org.onosproject.onos-api:1.10.0.SNAPSHOT]
    at org.onosproject.incubator.store.meter.impl.DistributedMeterStore.getAllMeters(DistributedMeterStore.java:241)[135:org.onosproject.onos-incubator-store:1.10.0.SNAPSHOT]
    at org.onosproject.incubator.net.meter.impl.MeterManager$InternalMeterProviderService.pushMeterMetrics(MeterManager.java:245)[133:org.onosproject.onos-incubator-net:1.10.0.SNAPSHOT]
    at org.onosproject.provider.of.meter.impl.OpenFlowMeterProvider.pushMeterStats(OpenFlowMeterProvider.java:255)[170:org.onosproject.onos-providers-openflow-meter:1.10.0.SNAPSHOT]
    at org.onosproject.provider.of.meter.impl.OpenFlowMeterProvider.access$100(OpenFlowMeterProvider.java:88)[170:org.onosproject.onos-providers-openflow-meter:1.10.0.SNAPSHOT]
    at org.onosproject.provider.of.meter.impl.OpenFlowMeterProvider$InternalMeterListener.handleMessage(OpenFlowMeterProvider.java:384)[170:org.onosproject.onos-providers-openflow-meter:1.10.0.SNAPSHOT]
    at org.onosproject.openflow.controller.impl.OpenFlowControllerImpl$OFMessageHandler.run(OpenFlowControllerImpl.java:758)[165:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745)[:1.8.0_111]
2017-03-03 05:55:07,642 | ERROR | of-event-stats-6 | OpenFlowControllerImpl           | 165 - org.onosproject.onos-protocols-openflow-ctl - 1.10.0.SNAPSHOT | Uncaught exception on onos-of-event-stats-6
org.onosproject.store.service.ConsistentMapException$Timeout: onos-group-store-keymap
    at org.onosproject.store.primitives.DefaultConsistentMap.complete(DefaultConsistentMap.java:228)[125:org.onosproject.onos-api:1.10.0.SNAPSHOT]
    at org.onosproject.store.primitives.DefaultConsistentMap.values(DefaultConsistentMap.java:143)[125:org.onosproject.onos-api:1.10.0.SNAPSHOT]
    at org.onosproject.store.primitives.ConsistentMapBackedJavaMap.values(ConsistentMapBackedJavaMap.java:141)[125:org.onosproject.onos-api:1.10.0.SNAPSHOT]
    at org.onosproject.store.group.impl.DistributedGroupStore.getStoredGroups(DistributedGroupStore.java:351)[129:org.onosproject.onos-core-dist:1.10.0.SNAPSHOT]
    at org.onosproject.store.group.impl.DistributedGroupStore.pushGroupMetrics(DistributedGroupStore.java:1322)[129:org.onosproject.onos-core-dist:1.10.0.SNAPSHOT]
    at org.onosproject.net.group.impl.GroupManager$InternalGroupProviderService.pushGroupMetrics(GroupManager.java:387)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
    at org.onosproject.provider.of.group.impl.OpenFlowGroupProvider.pushGroupMetrics(OpenFlowGroupProvider.java:231)[169:org.onosproject.onos-providers-openflow-group:1.10.0.SNAPSHOT]
    at org.onosproject.provider.of.group.impl.OpenFlowGroupProvider.access$100(OpenFlowGroupProvider.java:86)[169:org.onosproject.onos-providers-openflow-group:1.10.0.SNAPSHOT]
    at org.onosproject.provider.of.group.impl.OpenFlowGroupProvider$InternalGroupProvider.handleMessage(OpenFlowGroupProvider.java:328)[169:org.onosproject.onos-providers-openflow-group:1.10.0.SNAPSHOT]
    at org.onosproject.openflow.controller.impl.OpenFlowControllerImpl$OFMessageHandler.run(OpenFlowControllerImpl.java:758)[165:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
So, that’s a really good sign IMO! I think it’s just the transport now. No Copycat log issues, no Copycat client issues. Just seems to be the transport.
:clap:
wish I could track down a case of the incosistency issue to double check it but that would probably take an eternity
Jordan Halterman
@kuujo
Mar 03 2017 19:06
probably should have added a log message to the sequencer when it resets
Jon Hall
@jhall11
Mar 03 2017 19:09
:)
Jordan Halterman
@kuujo
Mar 03 2017 19:09
The clients’ KeepAliveRequests are causing the servers to resend missing events, but they’re also succeeding so the clients are not trying to find other servers. The PublishRequests just fail and the client waits forever for them.
If you look at server 3 for session=174435 you’ll see the client continue to submit KeepAliveRequest and get PublishRequests back right after that that are lost by the No handler issue. But KeepAliveRequest succeeds so the client stays connected to that server
so, fix that and we should be g2g
Jon Hall
@jhall11
Mar 03 2017 19:11
oh, I guess that issue is a more serious than I though
Since it doesn’t look like it recovers
Jordan Halterman
@kuujo
Mar 03 2017 19:11
yeah… previously for some reason clients seemed to manage to switch servers
but doesn’t seem this one is, which makes sense
it can’t know what it doesn’t know
Jon Hall
@jhall11
Mar 03 2017 19:12
:)
Jordan Halterman
@kuujo
Mar 03 2017 19:12
if its requests are succeeding then as far as it can tell it’s just not getting any events from the cluster
Jon Hall
@jhall11
Mar 03 2017 19:13
ok, let me take a spin with your transport changes
Jordan Halterman
@kuujo
Mar 03 2017 19:13
:+1:
Jon Hall
@jhall11
Mar 03 2017 19:13
and see if I can get that into the release
Looks good to me so far
but I didn’t look at that test yet
I don’t know if you saw, but the unit tests failed on that
Jordan Halterman
@kuujo
Mar 03 2017 19:17
damn hadn’t looked yet… hmmmmmm…..
oh I know why duh
just a race condition
one sec
Jordan Halterman
@kuujo
Mar 03 2017 19:30
ehh gotta rebase
Jon Hall
@jhall11
Mar 03 2017 19:31
thats my fault :)
Jordan Halterman
@kuujo
Mar 03 2017 19:33
alright pushed
Jordan Halterman
@kuujo
Mar 03 2017 19:46
That implementation should be pretty close to what's in Copycat itself. At least a lot closer and a big improvement. I'll have to actually look the Netty messaging service when I find some time to make sure they're totally consistent.
Jordan Halterman
@kuujo
Mar 03 2017 19:52
sweet that one worked
Jon Hall
@jhall11
Mar 03 2017 19:57
hmm, it looks like the nodes are never able to form a cluster with this
I think the heartbeats aren’t getting through
Jordan Halterman
@kuujo
Mar 03 2017 20:03
ugh I knew that was going to happen
Jordan Halterman
@kuujo
Mar 03 2017 20:13
Any useful log messages?
Jon Hall
@jhall11
Mar 03 2017 20:20
here are the logs
Jordan Halterman
@kuujo
Mar 03 2017 20:32
I see the problem
hmmmmm
I was able to reproduce it in a test
Jordan Halterman
@kuujo
Mar 03 2017 20:38
it’s because when the Mode is set to SERVER it wasn’t connecting, so when servers connected to other servers no connection would be created and messages would fail
Jon Hall
@jhall11
Mar 03 2017 20:38
ah
Jordan Halterman
@kuujo
Mar 03 2017 20:46
I tested it the best I could. I actually used the transport in Copycat. My test transport just sucks too much for it to actually work, but caught that issue and fixed it.
k pushed
:pray:
Jon Hall
@jhall11
Mar 03 2017 20:49
What is your environment for testing? It’s possible you can use a mininet script to test a small cluster
Jordan Halterman
@kuujo
Mar 03 2017 20:53
Yeah... well the problem is I no longer have my work laptop now, and haven't had time to set up a lot of things on my personal laptop because I've been working on the actual problems. My technology is currently in flux. Haha
I'm in between laptops :-P or something like that
Using my wife's laptop actually
Haha
Jon Hall
@jhall11
Mar 03 2017 20:53
:)
Jordan Halterman
@kuujo
Mar 03 2017 21:01
Fortunately I seem to have written some code on this thing at some point in the past. But if this one dies I'll be working on my phone :-D
Jon Hall
@jhall11
Mar 03 2017 21:04
that seems like it would be very painful
Jordan Halterman
@kuujo
Mar 03 2017 21:04
hahaha
Jon Hall
@jhall11
Mar 03 2017 21:05
but then, I don’t even like texting on my phone
Jordan Halterman
@kuujo
Mar 03 2017 21:05
I actually do that a bit. Well, I don't write real code, but I often respond to SO questions and Copycat/Atomix questions on my phone. It is miserable.
Stupid curly braces
Jon Hall
@jhall11
Mar 03 2017 21:16
hmm, it looks like only the last of 7 nodes fully connected this time
Jordan Halterman
@kuujo
Mar 03 2017 21:19
Okay well I'm probably just going to have to spend some time to set up an actual environment I can test this in then...
Jon Hall
@jhall11
Mar 03 2017 21:19
sure
Jordan Halterman
@kuujo
Mar 03 2017 21:41
hmm the transport actually does work in Copycat
…after I fixed the last issue I pushed
you can try it, but I’m not so sure that fixes the problem in ONOS… setting up some better tests anyways
dunno how long it will take me :-)
Jordan Halterman
@kuujo
Mar 03 2017 21:47
ugh virtualbox
Jon Hall
@jhall11
Mar 03 2017 22:14
hmm, it looks like starting the nodes in sequence makes the connection issues worse than starting in parallel
Jordan Halterman
@kuujo
Mar 03 2017 22:18
k let’s see if I got everything now...
Jordan Halterman
@kuujo
Mar 03 2017 23:46
Won't be able to test and fix it until this evening. Mininet is not cooperating with me for some reason