Anybody knows any relevance of Blox with another apache project called Apache Oozie?
Thanks for the response. I guess my question was from the standpoint of capabilities. I know Oozie is created to operate within the Apache Hadoop ecosystem and Blox may be more EC2 based workload focused. Since Blox is open source under apache license, I was of any relevance with Oozie. May it be any similarities or striking differences.
Hey @mkala1, good questions.
Oozie is really an orchestration framework. A common use case being, at 2am wake up and launch a MapReduce job, which then launches a Hive query, which then runs a bash script, which then sends out a completion email. Blox is a scheduling framework for Docker containers on ECS. A common use case being, schedule a metrics daemon across all instances in your ECS cluster. Blox would then monitor your metrics daemon ensuring it stays up, and automatically schedule the daemon as new instances come online.
So I would say they are similar in that they both schedule things. But Oozie is more focused on workflows, whereas Blox is more focused on launching long-lived Docker daemons right now.
Hey @dshin192. The data store (Etcd) is just where the cluster-state-service stores the local ECS state. As to your question about why cluster-state-service over calling ECS APIs directly, there are a number of reasons:
- The cluster-state-service consumes the ECS event stream instead of polling. This means it is in near real-time sync with changes to your ECS clusters.
- It exposes the ECS event stream via a REST API method that you can consume. For example, by consuming this streaming API, you would be notified immediately as a new ECS instance comes online or a task unexpectedly stops. No polling needed.
- Avoids any ECS API rate limits that you would probably run into if you were continuously polling everything every few seconds.
Let me know if you have any follow-up questions.
We've just released Blox v0.2.0. Check out https://github.com/blox/blox/blob/master/CHANGELOG.md for a list of new features and bug fixes included in this release.
We've just released Blox v0.2.1 with a number of important bug fixes. Check out https://github.com/blox/blox/blob/master/CHANGELOG.md for a list of bugs fixed in this release. We strongly encourage anyone running Blox v0.2.0 to update to v0.2.1.
We've just released Blox v0.3.0. A list of new features included in this release can be found at https://github.com/blox/blox/blob/master/CHANGELOG.md
@jicowan Good question. The Blox daemon-scheduler does not get any priority over tasks started by other schedulers. However, you could accomplish that by starting the daemon-scheduler before calling start task to start any additional tasks. Also, the daemon-scheduler is usually pretty quick to launch the task when a new instance comes online. So it will probably beat anything else for starting new tasks when new instances joins your cluster because it actions directly on the ECS event stream.
@kylbarnes it would be nice to have a mechanism to reserve the necessary resources to start the daemon container or set the start order for tasks. I'm concerned about when a container instance fails as the cluster is nearing capacity. If there are insufficient resources to reschedule the tasks from the failed container instance, the scheduler will put the tasks in a pending state until the new container instance comes online. In this condition, which tasks will be scheduled first? The pending tasks or the daemon?
@jicowan Can you elaborate on what you’re asking? If a container instance fails, any running tasks that were running on it are not automatically rescheduled. It is up to the scheduler that launched them to notice that they are no longer running, and reschedule them. The Blox daemon-scheduler does this. Also, the ECS service scheduler does this (launching a task by creating a service). But anything you launched by manually calling run-task or start-task does not automatically get restarted. So, assuming a container instance crashed and was restarted, the Blox daemon-scheduler is listening to the ECS event stream, and would launch tasks almost immediately after it comes back online. Is your question about how to dictate the order of what tasks the Blox scheduler launches in order? Or how to dictate the task launch order between multiple different schedulers, such as Blox daemon-scheduler vs ECS service scheduler? Also, please note that our AWS deployment of Blox has it running on it’s own dedicated cluster. So it wouldn’t contend for resources with other tasks being launched. However, you may run into this if you deploy Blox onto a shared cluster.
@kylbarnes Thanks for the clarification Kyle. I meant to say the scheduler will restart the tasks of your ECS service. I am really asking about how to dictate the task launch order between different schedulers. I know you say that the Blox daemon scheduler will quickly launch the daemon on a new container instance, but I'm concerned about when there are multiple schedulers all trying to schedule tasks on an instance at once. It would be nice to be able to assign a scheduler or task a priority value so it always starts before other tasks are scheduled on the instance. Or, if you can't assign a priority, then add the capability to reserve the resources (CPU, memory, and port) for the daemon so the daemon can always be scheduled on the instance.
Gotcha. I think I understand what you're asking now. So this is really around multiple schedulers competing for the same resources once an instance gets restarted. Since all schedulers will probably try to relaunch their own tasks as soon as the instance comes up. Knowing the way the Blox daemon-scheduler is written, it would probably be the first scheduler to launch tasks once the instance comes online, since it acts on the event stream, which is typically faster than other schedulers that act by polling. But I think you're asking in a more generic sense, right? How to guarantee that some portion of the container instances resources are reserved for a particular scheduler, regardless of the scheduler, correct? Or are you asking specifically about the Blox scheduler?
Gotcha. So I can tell you that it's not possible right now, since that would require the schedulers to orchestrate with each other, which different schedulers typically won't do. The only real practical way I can see to do this is to allow partitioning of the container instance resources into groups. And allow a scheduler to own a particular group. This is an interesting idea. I'll bring it up for discussion over here with our ECS PMs. You can also create a Blox issue for this, but I think this relates more to ECS than to Blox, since the resource partitioning would happen on the ECS side, not with the Blox daemon-scheduler.
Hi, i have install Blox in localhost, when i access http://localhost:2000, it showed 404, what the correct way to access it ?
@Tiny-wlx which API are you trying to call? You can either use the swagger client to not have to deal with constructing paths or depending on the API you have to construct the path like this http://localhost:2000/v1/environments (replace environments with the route of the API you want) . Here are the routes: https://github.com/blox/blox/blob/dev/daemon-scheduler/pkg/api/v1/router.go
Hi guys. I was curious if anyone can point me at the template and demo-cli mentioned in the post: https://aws.amazon.com/blogs/compute/introducing-blox-from-amazon-ec2-container-service/ They seem to be dead links. I would like try out the product in the easiest way possible. Thanks!
@nbrandaleone We've moved the 0.3 release into its own branch - v0.3. Here are links to the template(https://github.com/blox/blox/blob/v0.3/deploy/docker/conf/cloudformation_template.json) and the demo-cli (https://github.com/blox/blox/tree/v0.3/deploy/demo-cli). You can read more about the move on the readme and the faq (https://github.com/blox/blox/blob/dev/docs/faq.md) of the main repo.
scheduler_1 | 2017-10-04T15:08:20Z [CRITICAL] Could not start etcd: grpc: timed out when dialing
css_1 | 2017-10-04T15:08:19Z [CRITICAL] Error starting event stream handler: grpc: timed out when dialing
i'm getting one error on etcd_1 as well:
etcd_1 | 2017-10-04 15:08:13.321397 C | etcdserver: create wal error: rename /var/lib/etcd/wal.tmp /var/lib/etcd/wal: permission denied
which is weird
btw, i'm using the master branch
etcd_1 | 2017-10-05 11:29:06.271677 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd_1 | 2017-11-02 19:14:24.508051 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {"[::]:2379" <nil>}
etcd_1 | 2017-11-02 19:14:30.100438 W | etcdserver: apply entries took too long [29.34154ms for 2 entries]
etcd_1 | 2017-11-02 19:14:30.100516 W | etcdserver: avoid queries with large range/delete range!
etcd_1 | 2017-11-02 19:14:30.100438 W | etcdserver: apply entries took too long [29.34154ms for 2 entries]
etcd_1 | 2017-11-02 19:14:30.100516 W | etcdserver: avoid queries with large range/delete range!
Hi even am experiencing same issue
Any one using blox daemon-scheduler if so please let me know...
I am trying to write my own algorithm for placing containers in instances in ECS and trying to use Blox for this. But I am not able to understand where to write my code in Blox ?