funcatron/exchange

David Pollak

@dpp

Let's use the default namespace

The scripts/configs are starting points for folks

So, if there's a K8s guru admin person, she can modify the scripts to suit her needs

Emanuel Borsboom

@borsboom

Ok, so maybe I'll just prefix all the resources like rabbitmq with funcatron- to avoid confusion.

David Pollak

@dpp

Yeah... sounds right

For rabbit funcatron-rabbitmq

For the other components funcatron-frontend, funcatron-tron, funcatron-runner

Emanuel Borsboom

@borsboom

Oh... and it would be ideal if what we do runs on K8s 1.2 :-)

I think this will be doable, that's the release where resource types like Deployments and ConfigMaps were introduced.

The trick is finding a cluster that old :)

Emanuel Borsboom

@borsboom

Do you happen to have a v1.2 cluster running somewhere I can use to test? I tried with minikube but it doesn't support versions that old.

David Pollak

@dpp

I don't have one up right now. How about doing the PR against Minikube and I'll test against 1.2 when I can spin up a cluster.

Emanuel Borsboom

@borsboom

This looks like a decent approach to the "waiting for dependencies to be ready" problem in Kubernetes: https://blog.giantswarm.io/wait-for-it-using-readiness-probes-for-service-dependencies-in-kubernetes/

InitContainers are a nice newish feature in Kube 1.5, but this can be done without them too.

David Pollak

@dpp

Given that the thing the rest of the system needs to wait for is RabbitMQ and there's no HTTPish thing that I know of in Rabbit, I think the best thing to do is just start everything at once and I'll deal with it at the application level

Emanuel Borsboom

@borsboom

Ok, cool.

Emanuel Borsboom

@borsboom

@dpp: How do I tell tron which hostname to use to connect to rabbit? Currently it appears to be trying localhost:

Failed to open RabbitMQ Connection using {:hosts ["localhost"], :port 5672, :username "guest", :password "guest"}

Emanuel Borsboom

@borsboom

Point to Nginx port 80 and you should see it proudly saying "Funcatron"

Even though neither runner nor tron are running, the frontend still seems to be proudly saying "Funcatron", so not sure how much this test actually works out all the components.

David Pollak

@dpp

I'm going to have to update each of the components to use DNS to discover Rabbit... so there's no way things will "just work"... but as long as Nginx is running and you can see Rabbit and the JVM processes are running, I can do the rest.

Emanuel Borsboom

@borsboom

tron and runner both fail to start, so there are no JVM processes running in those.

David Pollak

@dpp

Are there logs for either?

Emanuel Borsboom

@borsboom

Yup, that's where I'm seeing the "Failed to open RabbitMQ Connection" message.

David Pollak

@dpp

Sorry... I'm blind and stupid :-(

Emanuel Borsboom

@borsboom

(so clearly a JVM process does start, it's just that it quits)

David Pollak

@dpp

1 sec... lemme see what to pass as the parameters to the JVM

Emanuel Borsboom

@borsboom

The mesos marathon config also has tron and runner request 2.0 CPUs and 2GB of memory, which is quite a lot (and means they won't run on the minikube's default settings). Should they really request that much?

David Pollak

@dpp

If you can pass --rabbit_host dns-name-for-rabbit to the tron an runner

1 cpu and 756MB should be okay

Emanuel Borsboom

@borsboom

Ok, would that go in the TRON_1 environment variable, or on the actual command-line?

David Pollak

@dpp

If you can pass it on the command line, that's easier

the TRON_1 env var was because I couldn't figure out how to pass variables to the JVM in Mesos

Emanuel Borsboom

@borsboom

Ah OK, I'll switch those all over to normal command-line

David Pollak

@dpp

and only Mesos knew the name and port of the Rabbit instance

Emanuel Borsboom

@borsboom

@dpp The marathon config has this for runner:

"acceptedResourceRoles": ["slave_public"]

The equivalent in Kube would probably setting a label on some nodes and then using a nodeSelector to schedule the pods only to nodes with that label. Should I set it up that way?

David Pollak

@dpp

Not if it requires extra work.

Emanuel Borsboom

@borsboom

(it would be up to the person deploying it to make sure there's a node with the right label)

David Pollak

@dpp

At this point, I care more about "making it run" than I do about specifics of where things are running.

Emanuel Borsboom

@borsboom

Ok, just trying to hew as closely to the mesos setup as possible :)

David Pollak

@dpp

There will likely be a future need for designating where things run, but not today.

I deeply appreciate your diligence. It's one of the reasons I love working with you!

Emanuel Borsboom

@borsboom

Ok, tron and runner are able to connect to rabbit. BTW Kube's default behaviour is to keep retrying starting the container if it crashes (with an increasing delay between retries), which ends up working pretty well to have tron/runner wait until rabbit is running.

@dpp: frontend looks like it depends on tron and rabbit (although it seems to start up just fine without them). How do I point it at the correct hostnames for those services?

Emanuel Borsboom

@borsboom

Also seeing a lot of these in frontend's log:

2017/04/10 23:46:10 [error] 7#7: *9460 connect() failed (111: Connection refused), context: ngx.timer

Also, should frontend be exposed outside the cluster (e.g. with a LoadBalancer service)?

David Pollak

@dpp

Yes... frontend should be exposed on port 80 outside the cluster

The connection refused thing... don't worry. I'll figure out how to get the Frontend to detect it's in a Mesos cluster and look for Rabbit

Emanuel Borsboom

@borsboom

Being able to pass in a hostname (e.g. via environment variable or otherwise) would probably be simplest. But I guess otherwise looking for a Service resource named funcatron-rabbit in the same namespace would be good.

Should rabbit have any "statefulness" (e.g. persistent storage) or clustering support?

Emanuel Borsboom

@borsboom

@dpp: first cut of Kube resources: funcatron/funcatron#21

Emanuel Borsboom

@borsboom

@dpp: I see you merged the PR. Anything else I can help with right now?

David Pollak

@dpp

@borsboom It's all good. Thank you for your help. I'll ping you when I need more. You and FP Complete are awesome!

Where communities thrive

People

Repo info

Activity