Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Janis Jansons
    @jansonz
    Hi guys. Is there a good way to set a variable in conductor in case it's not passes as workflow.input?
    10 replies
    I mean if workflow.input.variable2 is null use valueX (output from another task) as this variable?
    vwyuheng
    @vwyuheng
    ....
    hi
    ketantalreja
    @ketantalreja
    Hello All, I am using Conductor version 2.27.2 and now am planning to get all workflow execution metrics such as Task type, task input, task output, workflow status etc. from Conductor and have scraped to Prometheus server by which metrics can be shown or displayed in the front end, Is there an API or hook to get metrics from Conductor workflow executions? Has anyone integrated Conductor with Prometheus for metrics? Thanks!
    1 reply
    Bahadır Bozdağ
    @bahadirbb
    Hi everyone, we are planning to use Netflix Conductor as a real time workflow orchestrator. We are projecting 5000 workflows being started per second. Does anybody have experience on using Conductor with high load? And I suppose cassandra-persistence does not work out of box - do I need to implement a QueueDAO to use Netflix with Cassandra? or do you have any suggestions about high load?
    6 replies
    gnfpt
    @gnfpt
    Hi, I'm kind-of new to Conductor. I was wondering what would be the effort (if possible) to trigger workflows based on, for instance, a file in the file system instead of using the rest api. Any idea?
    17 replies
    dcore
    @dcore94
    Anyone has success experience with starting a SUB_WORKFLOW by passing the name of the sub workflow from a ${} expression? When I put the WKF name statically in the SUB_WORKFLOW task it works. When I link it to an expression I get "INTERNAL_ERROR - The Task ${anothertask.output.body.workflowname} defined as a sub-workflow has no workflow definition available". Is this supported by the way?
    5 replies
    dcore
    @dcore94
    In the meanwhile my python client which has been serving without issues for months suddenly started returning this message today upon updating a Task for completeion ... ERROR: {"code":"INVALID_INPUT","message":"INVALID_INPUT - key can't be empty","retryable":false,"instance":"bdf96d9d655b"}. Anynone able to point me at least to the code where this might happen? I'm pretty much clueless. The task status updated to "COMPLETED" (I see in the UI) but the workflow stucks there ...
    dcore
    @dcore94
    Found the log file with this ...
    [ERROR] 2020-09-02 09:11:05,984 WorkflowExecutor - Error deciding workflow: a1734b90-8128-4e45-93b8-6389c33bae97
    java.lang.IllegalArgumentException: key can't be empty
    at java.lang.System.checkKey(System.java:843)
    at java.lang.System.getProperty(System.java:716)
    at com.netflix.conductor.common.utils.EnvUtils.isEnvironmentVariable(EnvUtils.java:18)
    at com.netflix.conductor.core.execution.ParametersUtils.replaceVariables(ParametersUtils.java:210)
    at com.netflix.conductor.core.execution.ParametersUtils.replace(ParametersUtils.java:169)
    at com.netflix.conductor.core.execution.ParametersUtils.getTaskInputV2(ParametersUtils.java:125)
    at com.netflix.conductor.core.execution.ParametersUtils.getTaskInput(ParametersUtils.java:59)
    at com.netflix.conductor.core.execution.DeciderService.getTasksToBeScheduled(DeciderService.java:647)
    at com.netflix.conductor.core.execution.mapper.DecisionTaskMapper.getMappedTasks(DecisionTaskMapper.java:108)
    at com.netflix.conductor.core.execution.DeciderService.getTasksToBeScheduled(DeciderService.java:678)
    at com.netflix.conductor.core.execution.DeciderService.getTasksToBeScheduled(DeciderService.java:641)
    at com.netflix.conductor.core.execution.DeciderService.getNextTask(DeciderService.java:365)
    4 replies
    ketantalreja
    @ketantalreja
    Hi All, Does anyone has experience using netflix conductor with prometheus for monitoring? Or any thoughts on how to collect metrics from netflix conductor such as workflow and task executions etc using Prometheus. Thanks!
    32 replies
    Flávio Schuindt
    @flavioschuindt
    @kishorebanala One question for you. Are you guys on netflix using Dynomite with Redis or other persistent store for conductor? Reason I am asking is because I want to understand how Dynomite plays here with conductor. We have been using dynomite+redis and sometimes the communication conductor <-> dynomite simply stops to work and no matter what you do, it only comes back after you restart dynomite. This is very bad. Also, I see no activity in dynomite github project for more than one year which is not good, looks like a dead project. Do you have more documentation on HA part of conductor? I would like to understand how you guys are using dynomite (or other persistent store) in a way that works well without those disconnections and workflow losing execution track.
    5 replies
    ritu-p
    @ritu-p
    getStartDelayInSeconds
    does StartDelyinSeconds introduce a delay between the task start.I am trying to introduce a sleep between each loop of a DO_WHILE task.Is there any other way to achieve this.
    ritu-p
    @ritu-p
    Hi @kishorebanala Can you please take a look at the issue Netflix/conductor#1868 and the proposed solution?
    TIm
    @jinwik
    hi, @kishorebanala Do I need to open "workflow.decider.locking.enabled" when we use multi-instances conductor servers connecting with a single standard redis server in my local k8s,thx.
    3 replies
    vwyuheng
    @vwyuheng
    image.png
    1 reply
    dcore
    @dcore94
    Anyone tried to use Conductor as a coordinator for Map-Reduce based algorithms. Would it be viable?
    ritu-p
    @ritu-p
    @kishorebanala can you please take a look at this issue Netflix/conductor#1868 . I added the line eventTask.setCallbackAfterSeconds(taskToSchedule.getStartDelay()); but still it is triggering immediately the moment the first task is completed .Can you please suggest what can be done for my use case
    varchaswi gandham
    @varchaswi
    is there a auto clean up for conductor my ec2 shows 99% disk used and conductor docker wort restart after log files clean up , i see more lucene.dvd are they safer to deleted
    2) how do i skin it deploy as AWS EBS instead of in EC2
    3) any one has the git link to UI project other than to one form netflix
    Thanks
    muralicmr
    @muralicmr

    Hi All,

    If payload size is greater than 100KB, it is recommended to use s3 (https://netflix.github.io/conductor/bestpractices/). However, we are not planning to use s3. Are there any known issues if we use http task with a pay load of 1-3 MB directly without s3?

    2 replies
    sakella1986
    @sakella1986
    Hi All, quick question. https://netflixtechblog.com/netflix-conductor-a-microservices-orchestrator-2e8d4771bf40 states that "Amazon Step Function
    Recently announced AWS Step Functions added some of the features we were looking for in an orchestration engine. There is a potential for Conductor to adopt the states language to define workflows." is it still true?
    2 replies
    We are thinking of using ASL for building our own library for orchestration, or using conductor. Just want to understand the roadmap.
    Rahul Gupta
    @techyragu
    @all Hi, I recently merged the latest netflix conductor code in my local branch. But after merging , seems task is autmotically setting itself to completed status ? any idea?
    1 reply
    Raghuveer Kumarakrishnan
    @raghuftw
    Do folks run Conductor in production without configuring ElasticSearch? I am not keen on indexing, wonder if I can get by with the one running on localhost
    sakella1986
    @sakella1986
    @kishorebanala what are the minimum dependencies to install conductor? We don't use AWS, Elastic Search, Dyno etc.
    3 replies
    TajamulFazili
    @TajamulFazili
    Hi All,
    https://netflix.github.io/conductor/configuration/systask/#do-while-task says TaskType SUB_WORKFLOW is not supported as a part of loopover task , Could you please suggest any other way to achieve this.
    e.g Task1 output is a an array of strings and other workflow(sub workflow) accepts string as input. i want to start as many subworkflows as there are items in Task1 output array .
    1 reply
    Rahul Gupta
    @techyragu
    I am seeing lot of debug messages related to ConnectionPool? Is there any way to disable there debug messages?
    2020-10-12 08:48:53.595 DEBUG ConnectionPoolHealthTracker:93 - Running, pending pools size: 0
    2020-10-12 08:48:53.595 DEBUG ConnectionPoolHealthTracker:93 - Running, pending pools size: 0
    2020-10-12 08:49:03.595 DEBUG ConnectionPoolHealthTracker:93 - Running, pending pools size: 0
    2020-10-12 08:49:03.595 DEBUG ConnectionPoolHealthTracker:93 - Running, pending pools size: 0
    2020-10-12 08:49:13.596 DEBUG ConnectionPoolHealthTracker:93 - Running, pending pools size: 0
    2020-10-12 08:49:13.596 DEBUG ConnectionPoolHealthTracker:93 - Running, pending pools size: 0
    Rahul Gupta
    @techyragu
    retryDelaySeconds doesn't seem to be working. Task retries immediately without waiting for time specified in retryDelaySeconds
    1 reply
    Is there an issue existing related to this?
    Archana Pradeep
    @archthegit
    Hi @all I am using the go client and can't seem to figure out how it works. I have coded out my tasks and called client.Start() on both these tasks as shown in the sample programs but all i see is Polling for task: task_login with a: 10000 (ms) polling interval with 1 goroutines for task execution, with workerid as... How do i get the workflow running ?
    2 replies
    Archana Pradeep
    @archthegit
    Hi @all I am using the go client and have two task functions that I've started polling for, using client.start(). I have a workflow set up and running with these two tasks. However, I only see the first task go to "IN_PROGRESS" and then timeout. It looks like the task doesnt know it has to execute the function I've implemented, am I missing any step? I've registered my tasks and used the same task name as "task_type" in my client.start() function
    1 reply
    ketantalreja
    @ketantalreja

    Hi All, I am using Netflix conductor v2.27.2 with ES 6.4.3 configuration as below -

    workflow.elasticsearch.instanceType=external

    Transport address to elasticsearch

    workflow.elasticsearch.url=es:9300

    Name of the elasticsearch cluster

    workflow.elasticsearch.index.name=conductor
    workflow.elasticsearch.version=6
    workflow.elasticsearch.tasklog.index.name=task

    And I am getting error below when I am performing a search query on ES to get the total count for workflow executions,

    ERROR com.netflix.conductor.dao.es5.index.ElasticSearchDAOV5 - Failed to index object of type: task

    Here is the mapping information I get from /conductor/_mapping

        "conductor": {
            "mappings": {
                "workflow": {
                    "properties": {
                        "correlationId": {
                            "type": "keyword"
                        },
                        --------
                    }
                }
            }
        }
    }

    However, if I use ES 5.x with netflix conductor v2.27.2, there is no problem with the ES indexing on workflow exeuctions, and I get mapping information for both types i.e. workflow and task

    {
        "conductor": {
            "mappings": {
                "workflow": {
                    "properties": {
                        "correlationId": {
                            "type": "keyword"
                        },
                        --------
                    }
                },
                "task": {
                  "properties": {
                        "correlationId": {
                            "type": "keyword"
                        },
                        --------
                    }
                }
            }
        }
    }

    Could anyone help me with ES 6 index mapping issue I am facing and if I have missed anything in the configuration. As I understand, ES 6 module is already registered in netflix conductor v2.27.2 so I don't have to explicitly register in the netflix conductor.

    Thanks, Ketan.

    1 reply
    muralicmr
    @muralicmr
    Hi, Does any know if there is any plan to support java 11. I see that there was plan to do this in 2019 (Netflix/conductor#799). I get errors when I build conductor on Java 11.
    1 reply
    Archana Pradeep
    @archthegit
    Hi @all I am trying to create a decision task with a "caseExpression", however I get a "error while evaluating script: " . This is what the decision task looks like, can anyone help me figure out what is wrong ?
    "name": "task_check_balance",
    "taskReferenceName": "task_check_balance",
    "inputParameters": {
      "Amount": "${task_create_transfer_request.output.Amount}",
      "Balance": "${task_get_user_balance.output.res}"
    },
    "type": "DECISION",
    "caseExpression": "$.Amount > 0 ? 'Insufficient' : 'Sufficient'",
    "decisionCases": {
      "Sufficient": [
        { …..
    }
    ]
    4 replies
    ketantalreja
    @ketantalreja
    @all, @kishorebanala, could you please have a look into this issue's here Netflix/conductor#1925, Netflix/conductor#1926 and help in providing your inputs. Thanks!
    dcore
    @dcore94
    Hi all, it's been one week that we are trying to make conductor run with a docker-swarm configuration. We have several problems that we are not able to wrap our head around. Has anyone some experience or pointers to documentation for this? Thank you.
    12 replies
    Nilesh Sargar
    @nsargar
    Hi guys, I am new to Netflix Conductor, needed some info around property “conductor.queue.size” and what should be the value for production
    1 reply
    sakella1986
    @sakella1986
    does conductor support workflow/statemachine inheritance?
    1 reply
    franz040187
    @franz040187
    Hi guys, is anyone using conductor server in ssl? I have to expose conductor API in https, but, from configuration, i am able to change only port. Is there a way to go over https without customizing JettyServer classes? I have to do this also for UI. Moreover i am trying to add oauth2 strategy for logging in to UI (invoking wso2 server), but i have some problem to understand how to set returned token in cookie or session. Has anyone any ideas? Thanks, i am new with conductor, sorry for the questions. Thanks to all.
    Archana Pradeep
    @archthegit
    Hi @all I've been using conductor to run a very simple workflow that calls functions with no logic just return statements. But just running 4 such tasks in a workflow takes a good 2 seconds. How can I optimize the execution time? I've tried to reduce the polling interval but it doesnt seem to reduce the execution time of the workflow by much (get 1.8 seconds on avg)
    Nilesh Sargar
    @nsargar
    @franz040187 this is one of the property i have got from Netflix Conductor admin /config api, I mean the configuration Netflix conductor running on. my guess was Netflix conductor using this property for the scheduling queue (correct me if i am wrong) any comment on this property will be helpful. I am currently debugging an issue where in load, some workflow system tasks are stuck in scheduled state and some are not even being scheduled
    1 reply
    Jamie DeMichele
    @james-deee
    Hello, is there a roadmap or some kind of design document about what Conductor v3 will look like? Is development for v3 already under way? What does the expected upgrade path look for people going from v2 -> v3? Thanks for any info you've got :)
    Nilesh Sargar
    @nsargar
    @franz040187 btw we are using “redis” as db and not dynomite
    Juan Karsten
    @neophytecoder
    Hi guys, I've been using conductor with a service that takes long to respond(around 5 minutes). I set up the task definition to have zero retryCount. I want this task to be called only once. However, it seemed like conductor attempted to call the service several times with the same request and pollCount is incremented. Does anyone experience the same?
    "taskDefinition": { "name": "step", "retryCount": 0, "timeoutSeconds": 60000000, "timeoutPolicy": "TIME_OUT_WF", "retryLogic": "EXPONENTIAL_BACKOFF", "retryDelaySeconds": 60000000, "responseTimeoutSeconds": 60000000, "pollTimeoutSeconds": 60000000 }
    1 reply