by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Sabby Anandan
    @sabbyanandan
    Yikes. That still doesn't show why it is timing out. The SchedulerTaskLauncher is attempting to launch a task using SCDF's REST API, and for some reason in your K8s box, the API invocation is not going through intermittently. Networking-wise, something is odd.
    Siddhant Sorann
    @siddhantsorann
    Aaah. Understandable, but what's the default timeout on the scheduler for the SCDF API?
    Sabby Anandan
    @sabbyanandan
    FYI. Starting from 2.5, we have moved away from SchedulerTaskLauncher, so there's 1 less resource (deployment + pod) in K8s to deal with when scheduling tasks. You may want to upgrade to simplify all this.
    Siddhant Sorann
    @siddhantsorann
    Because as I can see here between launching the task and the error log only a second has elapsed. In general even when triggering via UI it takes more than a second to launch a task. Any way to increase this?
    Sabby Anandan
    @sabbyanandan
    There's no timeout configuration around SCDF's APIs. With the minimal information from the logs (and not knowing more about your environment), I'd check whether your ingress gating the SCDF's API is set up correctly. Perhaps also look at the access logs at the gateway to see if that gives any more details behind GATEWAY_TIMEOUT response.
    Siddhant Sorann
    @siddhantsorann
    @sabbyanandan Would love to upgrade! But we run more than a 400 task every day, and recently upgraded to 2.4.0 from 1.2.0 so another upgrade means another round of testing, so would need a solution for this for the current version itself.
    Alright, I'll look into that. If you get anything else on this, please do let me know.
    ab48917
    @ab48917

    Hmm… I think i see the hiccup. Its benign. Basically SCDF is trying to get the properties from the docker image and it can’t. I’ll open an issue for it.

    Thanks @cppwfs

    Siddhant Sorann
    @siddhantsorann
    @sabbyanandan One last thing, so the SchedulerTaskLauncherApplication service basically triggers the task on SCDF server using REST API, right?
    Sabby Anandan
    @sabbyanandan
    Correct.
    Siddhant Sorann
    @siddhantsorann
    @sabbyanandan Understood. I'll check in with our devops team for this. Seems like an ingress level issue. Thanks for the help!
    RaickyDerwent
    @RaickyDerwent
    Hi Guys!
    How do I pass parameters to a task schedule using the REST API ?
    I see two query params for properties and arguments, but none for parameters
    RaickyDerwent
    @RaickyDerwent
    Say I wanted to pass deployer.<app_name>.kubernetes.deploymentServiceAccountName and a couple of other environment variables, from the GUI I would just have to set it using parameters
    How do I do the same using the REST API ?
    RaickyDerwent
    @RaickyDerwent
    Nevermind, found it! Comma delimiter
    sumit1912
    @sumit1912
    Hi @sabbyanandan , I have created a crawler that fetches data from a DB and uploads to another service in batches(say 100). Everything works fine when ran in my local setup as a Spring boot app. I converted this to a SCDF Task service and executed it, it ran fine for some time and fetches/uploads data but after some time the DB connection is lost.
    When I ran the crawler with larger batch size(say 1000) so that the service completes fetching data faster the SCDF Task works fine and completes successfully.
    I suspect somehow the SCDF Task stops executing/breaks after executing for some default time. Is there some property that needs to be set in SCDF Task to let it execute for longer time? Any hint or suggestion will be of great help. Thanks in advance :)
    Glenn Renfro
    @cppwfs
    @sumit1912 Task doesn’t have a timer for database connections. It could be the datasource connection timeout (for example spring.datasource.hikari.connectionTimeout) or a network/firewall connection timeout.
    ab48917
    @ab48917
    @cppwfs - Any update on composed task runner throwing illegealArgumentException ? If you run a composed task runner and if any of tasks execution fails and after that trying to relaunch the same composed task than it is failing sometime with above exception and sometimes it is getting launched. Once it is failed than It is never coming up and we need to delete and re create the composed tasks again.
    Glenn Renfro
    @cppwfs
    I have not been able to recreate the problem yet.
    What database are you using?
    Can you provide a full log?
    ab48917
    @ab48917
    @cppwfs - I am using Oracle 12c. I will share you the logs.
    Do you want me to put here or should I send you on email?
    Glenn Renfro
    @cppwfs
    I think you can drag and drop it to gitter
    ab48917
    @ab48917
    Okay
    ab48917
    @ab48917

    @cppwfs I am triggering the task from SCDf and expecting to pick the volumeMount which I have provided in server-deployment.yaml but Task is not picking the volume mount as default mentioned on server-deployment .
    server-deployment.yaml has below configuration>

    spec:
    containers:

      - name: scdf-server
        image: springcloud/spring-cloud-dataflow-server:2.6.0-RC1
        imagePullPolicy: Always
        volumeMounts:
          - name: vol-arg
            mountPath: /mnt/scdf/
            readOnly: true
          - name: database
            mountPath: /etc/secrets/database
            readOnly: true
    ab48917
    @ab48917
    @cppwfs - is there any way to define the task volumeMount while spinning the pod through SCDF 2.6.0.RC1?
    Glenn Renfro
    @cppwfs
    ab48917
    @ab48917
    @cppwfs - I tried that volumeMounts by passing the deployer properties while launching the composed task but It's not taking
    Siddhant Sorann
    @siddhantsorann
    Hey guys, me and a couple of people from my team would like to contribute to the SCDF GitHub project. Can you suggest to us the best way to go about this?
    kalkir
    @kalkir
    hi, have installed scdf, but seeing 405 response while accessing streams & runtime from dashboard. skipper server logs says:
    WARN 1 --- [nio-7577-exec-5] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'GET' not supported
    any help would be appreciated. thanks
    Glenn Renfro
    @cppwfs
    @ab48917 you can open a github issue on SCDF for this issue.
    kalkir
    @kalkir
    Okay, thanks.
    ab48917
    @ab48917

    @ab48917 you can open a github issue on SCDF for this issue.

    Please provide the template/details to raise a JIRA for this? and JIRA board details as well.

    Sabby Anandan
    @sabbyanandan
    @siddhantsorann: Excellent! Feel free to choose existing issues with contributions welcome label. Otherwise, if you have any particular area where you feel it needs improvement, you can open a new issue and work on them, too. The SCDF UI project is another area you can look for enhancements also.
    @ab48917: We don't use JIRA. Feature requests and bug-fixes are directly handled as GH issues. Example: https://github.com/spring-cloud/spring-cloud-dataflow/issues
    Siddhant Sorann
    @siddhantsorann
    Thanks @sabbyanandan we'll start exploring those.
    Felipe Gutierrez
    @felipeg48

    Hi guys I have a custom source app, is a web app.
    I want to use the Metrics, prometheus and grafana.
    Im adding these to my custom Stream (web app)

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>
    <dependency>
        <groupId>io.micrometer.prometheus</groupId>
        <artifactId>prometheus-rsocket-spring</artifactId>
        <version>1.0.0</version>
    </dependency>

    When deploy my Stream Im using the following properties:

    app.movie-web.management.metrics.export.prometheus.enabled=true 
    app.movie-web.management.metrics.export.prometheus.rsocket.enabled=true 
    app.movie-web.management.metrics.export.prometheus.rsocket.host=prometheus-rsocket-proxy 
    app.movie-web.management.metrics.export.prometheus.rsocket.port=7001

    Exposing the movie-web app the /actuator doesn't show the /actuator/prometheus so
    it's not being displayed in the Grafana dashboard.

    Am I missing something??
    The auto-configuration should pick everything up and enable prometheus endpoint right?

    My stream is defined like this: movie-web | splitter | groovy-transform | jdbc but only the app-starters are showing up in the grafana dashboard, but no the movie-web
    Sabby Anandan
    @sabbyanandan
    Hi, @felipeg48. I believe you are missing the app-starters-micrometer-common dependency on your custom application. You can use one of the monitoring samples to follow-along and apply the similar dependency chain to get it up and running with Prom + Grafana.
    Felipe Gutierrez
    @felipeg48
    Thanks @sabbyanandan I’ll try that
    ab48917
    @ab48917
    Hi @sabbyanandan & @cppwfs - Two issues with composed task runner 2.6.0.RC1 -
    1- When you run a composed task , It is sometimes creating the black app execution .
    2- It is getting hung at any app level and not moving on. Neither it is saying anything or any error in SCDF logs. It is also not starting the app so we can see the logs at app level. Atttached screen print for two composed task to show both the cases.
    image.png
    #4950 - It is generated blank however, It should start the #4951. After sometime, It initiated the 4951 but it is never triggered the task - tsdl.
    Siddhant Sorann
    @siddhantsorann

    Hey @sabbyanandan For the gateway timeout issue, I changed the ingress Nginx timeout to 3mins. Still getting the following issue intermittently in scheduler service -
    Also, the pods which get this issue return an error in about 20 seconds.

    Caused by: org.springframework.cloud.dataflow.rest.client.DataFlowClientException: GATEWAY_TIMEOUT
        at org.springframework.cloud.dataflow.rest.client.VndErrorResponseErrorHandler.handleError(VndErrorResponseErrorHandler.java:74) ~[spring-cloud-dataflow-rest-client-2.4.0.RELEASE.jar!/:2.4.0.RELEASE]

    Also in our SCDF Server logs, we are getting this issue intermittently as well -

    2020-08-06 09:01:54.965  WARN 1 --- [-nio-80-exec-10] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: null
    2020-08-06 09:01:54.966 ERROR 1 --- [-nio-80-exec-10] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-1 - Connection is not available, request timed out after 30000ms.
    2020-08-06 09:01:54.969 ERROR 1 --- [-nio-80-exec-10] o.s.c.d.s.c.RestControllerAdvice         : Caught exception while handling a request
    
    org.springframework.dao.DataAccessResourceFailureException: unable to obtain isolated JDBC connection; nested exception is org.hibernate.exception.JDBCConnectionException: unable to obtain isolated JDBC connection
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:277)

    Somehow it's unable to connect to DB. Do you think these could be related? Or any advice here?

    Sabby Anandan
    @sabbyanandan

    @siddhantsorann: Ah, possibly related. You can change the datasource settings to have a bigger timeout value, but you may want to check why the DB connection is unstable.

    It'd really also help to understand what is spitting out GATEWAY_TIMEOUT and why; perhaps there's more stacktrace to it with the DEBUG enabled for Spring's REST template.

    ab48917
    @ab48917

    Hi @sabbyanandan & @cppwfs - Two issues with composed task runner 2.6.0.RC1 -
    1- When you run a composed task , It is sometimes creating the black app execution .
    2- It is getting hung at any app level and not moving on. Neither it is saying anything or any error in SCDF logs. It is also not starting the app so we can see the logs at app level. Atttached screen print for two composed task to show both the cases.

    @cppwfs - Can you please advise me here? is it a know issue or we can log a dehect for this?

    Sabby Anandan
    @sabbyanandan

    @ab48917: First, please switch to use 2.6 GA release; it went live last week. Secondly, there are a lot of variables that we don't have visibility into. For example, where you running, which platform, how you running, and what apps are in the CTR, etc.

    It is hard to grok it in an ad-hoc chat format, without much context. I'd suggest writing the step-by-step details either in a gist or as a README in your GH repo. We can review your setup and attempt to reason through what could be causing the hold up in your box.

    ab48917
    @ab48917
    @sabbyanandan I have deployed the SCDF2.6.0 in Openshift with K8s 1.11 version by creating the image of spring cloud dataflow jar into image. I have injected the composed task runner by setting the server-deployments.yaml file configuration from outside.