"while one is currently running" <- I missed this on your previous post. What I shared before was around launching tasks with different args/props, and that is the expected behavior. It continues to work that way.
However, there's now governance around task launches in 2.3. We use the task definition to parse the app, its version, args, and props. Based on whatever that is changed, we apply that on new launches. It can be app-version or args or props -- all of it. You get to practice CICD on task apps with it. With that, there is now the validation to make sure the task definition is currently not running.
If I have to restate your requirement, you want to launch multiple versions of the same task-definition and concurrently with different args. Yeah?
localscheduler support is not available yet. You can try to see if this PR can help your case: spring-cloud/spring-cloud-scheduler-quartz#1
Also, is there a reason that the scheduling agent was created to trigger tasks via rest api in 2.3.0? Previously the cronjob created in K8 was directly running the tasks.
@siddhantsorann: You may want to review the new CICD support for Tasks: https://docs.spring.io/spring-cloud-dataflow/docs/2.3.0.RELEASE/reference/htmlsingle/#spring-cloud-dataflow-task-cd
When the scheduler is used in K8s, to support CD flows for tasks, we needed a consistent mechanism to determine whether or not a given task is already running.
The only approach to determine that is to ask SCDF itself, so the new
scheduler-tasklauncher app is spawned behind the scenes (similar to composed-task-runner) to query SCDF at runtime, and then intelligently decide whether to consume new versions of the task apps or its property changes.
As far as I can see the PR has been waiting since over a year to be merged. Is there a specific reason for keeping that merge request open for such a long time?
@kasim-ba: It is a common practice for folks to rely on an enterprise scheduler in a production setting. In K8s, we rely on the scheduler agent in Kubernetes itself — assuming users would use a production-ready K8s cluster. Likewise, in CF, we rely on
pcf-scheduler, which is a managed service in Cloud Foundry.
However, in local, there's no platform-like concept attached to it. SCDF is a Boot app that runs as a local Java process in the VM or in the Docker Deamon (if you're using Docker Compose).
All that said, the quality of service and resilient operations are limited in the local method. If something goes wrong or if SCDF app crashes, there's nothing to bring it back up automatically. It is your responsibility to manage and operate it reliably — that's why it is typically used for local development.
Given these limitations, we didn't want to take upon the local-scheduler in our releases. If you're going to need a Scheduler in local-mode, it is a common practice to use quartz-based implementation locally. It will be a simple Boot app, too. There are many Boot + Quartz examples you can find online.
@/all Hi, folks! Thank you for your interest and contributions to Spring Cloud Data Flow and the ecosystem. We would love to learn more about your feedback and/or concerns, so please take this 1-page survey to let us know: https://twitter.com/pivotal/status/1204532909065080842
ps: I posted a similar message in the Spring Cloud Stream Gitter channel, so if you have feedback at the framework level, we would appreciate it if you can also complete the other survey.