These are chat archives for atomix/atomix

27th
Apr 2017
Jordan Halterman
@kuujo
Apr 27 2017 10:27
@txm119161336_twitter I didn't say it's not important. I said it's not affecting me at the moment, meaning nobody has filed a Jira issue complaining about our cluster being blocked on reconfiguration :-P I just have higher priority issues to address ATM. Namely, improving the stability of Copycat overall, which we've put a lot of work into over the last few months. We do support/use cluster configuration changes, but they simply aren't used enough for it to have become an issue yet. Or, perhaps more accurately, our QA team hasn't tested them enough for it to become an issue yet? I do think it's an important issue, but time is finite, and general fault tolerance issues is what people bug me about on a daily basis.
Jordan Halterman
@kuujo
Apr 27 2017 10:42

@johnou that seems to be the problem users of ONOS are having too, and I'm trying to figure out a way to prevent it. So, I just merged the first change to the ONOS threading model. We use Gerrit, but I still do development in GitHub PRs on my fork. Here's the PR that was ultimately merged: kuujo/onos#2

That PR actually doesn't have the final threading model in it though. After having a discussion yesterday, I decided that approach wasn't quite sufficient. Basically, ONOS has a bunch of primitives that are basically way better versions of Atomix's resources (and will later be merged into Atomix). That PR really just assigns a single logical thread to each partition of each primitive. But if one blocks the thread inside a primitive to make another call on the same primitive, the future will still never be completed.

So, what I really think needs to be done for total stability is a lot more complicated than that. The challenge is, we need to preserve order as much as possible. 50% of what consensus provides is order. We need to make sure a client that's receiving e.g. lock events always sees a "lock" event before "unlock" and never sees an unlocked lock being unlocked. So, that means we can't use thread pools. If we use thread pools then the OS can schedule the threads in any order it wants, and we can't guarantee the ordering of events.

The problem we've been seeing in ONOS is because there are so many threads, it's easy for some thread to block inside an I/O thread and cause other threads' requests to timeout. So, adding more threads by using a thread-per-partition-per-primitive added threads to make that less likely to happen, but it's still possible.

The ideal solution is a threading model that can provide the same ordering guarantees but handle arbitrary blocking of threads. So, for each partition of each primitive, it would have its own ordered Executor that runs on a shared thread pool but detects blocked threads and uses backup threads when necessary. If thread A is blocked, thread B is used. If thread B is blocked, thread C is used, and so on. When thread B because unblocked, the next callback is run on that thread assuming thread A is still blocked. The oldest threads are prioritized. That is essentially best effort ordering that ensures we can still complete futures on a thread that doesn't affect the Copycat client and users can arbitrarily block, which we've decided they should indeed be able to do.

So, that threading model is what I'm working on now and will be submitted to ONOS in a separate patch. It's admittedly difficult to do. I wanted to try to use Executors, but there's always a race between submitting a callback and determining whether that callback is going to block the thread. So, I have to implement the queueing and threading myself to detect blocked threads when tasks are pulled from the queue rather than when they're submitted. It can still be done with Executors, but some tasks may just be no-ops, and I'm not sure how I feel about that yet.

Hmm... I think I actually made myself feel a little better about it :-) I guess I'll see what I feel comfortable with tomorrow
Johno Crawford
@johnou
Apr 27 2017 12:19
@kuujo yeah certainly an interesting and important subject, it's dire that the developer knows which thread will run specific code, we have a few safe guards in our application to avoid that at runtime.. for example it's easy to block database calls from netty threads which in our case are blocking io calls, just configure Netty with a custom thread factory that sets a boolean in ThreadLocal and then check that with an aop pointcut configured for daos
public class NonBlockingIOThreadFactory extends DefaultThreadFactory {

    public NonBlockingIOThreadFactory(String poolName) {
        super(poolName);
    }

    @Override
    public Thread newThread(Runnable r) {
        return super.newThread(() -> {
            BlockingIODetector.preventBlockingIOinThisThread();
            r.run();
        });
    }
    ...
}
public class BlockingIOInterceptor implements MethodInterceptor {

    @Override
    public Object invoke(MethodInvocation methodInvocation) throws Throwable {
        BlockingIODetector.beforeBlockingIO(methodInvocation.getMethod());
        return methodInvocation.proceed();
    }
}
Johno Crawford
@johnou
Apr 27 2017 12:26
yeah ordering is a pain without introducing a bottleneck, orbit execution class might be interesting to you, feedback would be most welcome
https://github.com/orbit/orbit/blob/master/actors/runtime/src/main/java/cloud/orbit/actors/concurrent/WaitFreeExecutionSerializer.java
waitFreeExecutionSerializer.png