by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Sabby Anandan
    @sabbyanandan
    I'm glad with extra quotes, you're able to get everything up and running.
    Doug Whitehead
    @dsw9742
    is 2.0.0.M2 still scheduled to be released this week? if so is there a target date that can be shared?
    Sabby Anandan
    @sabbyanandan
    Hi, @dsw9742. Yes; we are targeting to wrap it this week. You can follow-along here: https://github.com/spring-cloud/spring-cloud-dataflow/milestones
    Doug Whitehead
    @dsw9742
    that is great, thanks @sabbyanandan
    Nicolas Widart
    @nWidart

    hello, I've been getting the following error and not sure what's causing it 🤔Dependencies were updated but that's it.

    The bean 'transactionManager', defined in org.springframework.cloud.task.configuration.SimpleTaskConfiguration, could not be registered. A bean with that name has already been defined in class path resource [org/springframework/batch/core/configuration/annotation/SimpleBatchConfiguration.class] and overriding is disabled.

    spring.main.allow-bean-definition-overriding is already set to true, yet the action mentioned is setting that to true

    Glenn Renfro
    @cppwfs
    Is SCDF reporting this exception? Or is your task application?
    Sabby Anandan
    @sabbyanandan
    @nWidart: Perhaps it'd be good to repost it at https://gitter.im/spring-cloud/spring-cloud-task.
    Nicolas Widart
    @nWidart
    The task application indeed, sorry, will post there :)
    djgeary
    @djgeary
    Hi @sabbyanandan yes we have it working now, and we have a way to modify our existing streams to get them to display in the dashboard. Everything works from the shell though without adding the quotes - the streams create and deploy successfully which implies to be its just a bug in the dashboard display? At least it should be consistent - currently you can create and deploy streams from the shell and not realise there is an issue until you try to view them in the dashboard.
    Another issue we encountered with the upgrade was the new concurrent task executions limit functionality. We initially were unable to launch tasks and with the error message 'The maximum concurrent task executions [20] is at its limit.' Looking into this SCDF calculates the number of running tasks to be the number of tasks in the execution table with a null end_time. For various reasons (eg tasks that died or were killed by kubernetes) we had a few hundred entries in here with null end_times. We've worked around this for now by setting spring.cloud.dataflow.task.maximum-concurrent-tasks to a very high value effectively disabling this feature, but I'm not sure using end_time=null is a reliable way to determine the number of concurrent tasks?
    Sabby Anandan
    @sabbyanandan

    @djgeary: Thanks for the feedback. I will look into (1) from the UI perspective. For (2), though, when we implemented it, we didn't have a way to determine whether if the Task is running, and its current status apart from inspecting the start < - > end times. For situations that aren't in our control (such as a pod crash), I can see how it would trigger a false alarm.

    Feel free to drop an issue for both of them, and we will review it on our side. If you have any suggestions on the solution, please have at it as PRs.

    Doug Whitehead
    @dsw9742
    Is there any way to get the UI to display "complete" stream graphs composed of multiple streams that start or end with taps? Obviously not having this doesn't "break" anything but it sure is a lot easier to see inter-stream relationships
    Is there any way to get the UI to display "complete" stream graphs composed of multiple streams that start or end with taps? Obviously not having this doesn't "break" anything but it sure is a lot easier to see inter-stream relationships
    Sabby Anandan
    @sabbyanandan

    If I understand your question correctly, you'd like to see the connected graph of all the streams that are attached to a "primary pipeline" via TAPs. Did I get that right?

    If yes, there's such option from the stream-list page. If you click the i icon of the "primary pipeline", it'd go to the stream-details page. In that page, you'd see Graph tab, which will show all the TAP connections associated with it.

    For example:

    timer = time --date-format=hhmmss | log
    minutes = :timer.time > transform --expression='payload.substring(2,4)' | log
    seconds = :timer.time > transform --expression='payload.substring(2)' | log

    The timer is the "primary pipeline", so after when these streams are created, you can click the i icon of timer stream from the list page to view the connected graph.

    Here's how it looks in the Dashboard:
    image.png
    Doug Whitehead
    @dsw9742
    thanks @sabbyanandan
    I'll have to play with this to see if I can reproduce
    Doug Whitehead
    @dsw9742
    I may be using destinations instead of taps ...
    Sabby Anandan
    @sabbyanandan
    If the destinations map to a primary pipeline name, you'd be able to automatically see the connected graph at the primary pipeline level.
    Doug Whitehead
    @dsw9742
    @sabbyanandan does that apply for streams that sink into a Router stream app that employs dynamically-generated destinations?
    Sabby Anandan
    @sabbyanandan
    No, the dynamic destinations are runtime resolved; we won't know what they would be beforehand, though.
    Doug Whitehead
    @dsw9742
    ah, got it
    rshailu
    @rshailu
    How do we configure Kafka with Spring Cloud Dataflow tile service as a substitute message service?
    Sabby Anandan
    @sabbyanandan
    Hi, @rshailu. Are you using the SCDF Tile for PCF?
    rshailu
    @rshailu
    yes
    Sabby Anandan
    @sabbyanandan
    Thanks. Check out the docs as to how to configure it.
    rshailu
    @rshailu
    Yeah, I have seen this but it has configuration for RabbitMQ but I am looking for Kafka
    Sabby Anandan
    @sabbyanandan
    Well, you'd exactly the same but with Kafka creds, of course.
    rshailu
    @rshailu
    Let me try it out
    Sabby Anandan
    @sabbyanandan
    Also, refer to Spring Cloud Stream docs.
    Doug Whitehead
    @dsw9742
    Parameter 2 of method functionSupport in org.springframework.cloud.stream.function.FunctionConfiguration required a bean of type 'org.springframework.cloud.stream.converter.CompositeMessageConverterFactory' that could not be found.
    getting the above error message when Spring Cloud Stream app deploys on Dataflow
    shouldn't CompositeMessageConverterFactory bean be created by the spring-cloud-stream dependency?
    I'm confused why it isn't there ...
    Sabby Anandan
    @sabbyanandan
    Please repost it at https://gitter.im/spring-cloud/spring-cloud-stream cc:/ @olegz
    Doug Whitehead
    @dsw9742
    thanks @sabbyanandan
    Justin Fleck
    @jfleck1_gitlab
    Hey all, we're trying to load scdf apps from a local nexus server that has SSL enabled on it with a self-signed certificate. We've tried a number of things with trust stores and are coming up short. What is the recommended way of adding a custom certificate, or custom ca chain, to SCDF's runtime so that maven and docker app pulls can work over SSL?
    Justin Fleck
    @jfleck1_gitlab
    This is our compose snippet so far (will eventually be in k8s but are trying to get it solved here first)
      dataflow:
        image: springcloud/spring-cloud-dataflow-server:2.0.0.BUILD-SNAPSHOT
        container_name: dataflow
        ports:
          - "9393:9393"
        environment:
          - spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers=kafka:9092
          - spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.zkNodes=zookeeper:2181
          - spring.cloud.skipper.client.serverUri=http://skipper:7577/api
          - JAVA_TOOL_OPTIONS="-Djavax.net.ssl.trustStore=/certs/truststore.jks -Djavax.net.ssl.trustAnchors=/certs/truststore.jks -Djavax.net.ssl.trustStorePassword=password123 -Djavax.net.ssl.trustStoreType=JKS -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true"
        depends_on:
          - kafka
        volumes:
          - "~/.m2:/root/.m2"
          - "${PWD}/certs:/certs"
          - "${PWD}/config/dataflow-config.yml:/application.yml"
    Justin Fleck
    @jfleck1_gitlab
    After trying to load bulk load apps from the nexus with a url like https://nexus.my.domain.com/nexus/repository/releases/com/domain/my/apps/0.0.5/apps-0.0.5.properties
    we get a large stacktrace in the dataflow log where the last causing execption is this;
    dataflow      | Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    dataflow      |     at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
    dataflow      |     at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120)
    dataflow      |     at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104)
    dataflow      |     at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:89)
    Justin Fleck
    @jfleck1_gitlab
    We're thinking about running our own certificate authority to actually be able to anchor the trust chain but.... yeah. Initial googling says it's an openjdk thing and to install the sun jdk. I can make my own Dockerfile but I'd like to stick with the supplied one as long as I can.
    So I was just wondering if there was a better way or should I continue down this path... as dark and scary as it may be
    Sabby Anandan
    @sabbyanandan
    Hi, @jfleck1_gitlab. I'm not very familiar in this subject, but the general recommendation is that you use a real-cert. The self-signed ones are typically used only in development environments, and you'll have to deal with some workaround or the other to get things line-up correctly. Sorry, I can't think of anything else.
    @/all FYI: SCDF v2.0 M2 is released -> https://spring.io/blog/2019/02/06/spring-cloud-data-flow-and-skipper-2-0-m2-released - please upgrade and let us know if you have any questions/feedback.
    Nicolas Widart
    @nWidart
    @sabbyanandan Awesome! I'm wondering, no mention of task metrics yet?
    Sabby Anandan
    @sabbyanandan
    Hi! That is not in scope for 2.0, at least as far as the Grafana dashboard + UI integration is concerned; we will follow-up on 2.0.1 and future incremental releases.
    Nicolas Widart
    @nWidart
    Oh ok :(
    Sabby Anandan
    @sabbyanandan
    One step at a time, but we will get there. Nothing really stops the community to jump-in and help us out, though. :smile:
    rshailu
    @rshailu
    Can someone guide me how to configure PCF Services with SCDF stream apps?
    I tried to use SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_SERVICES as deployment properties and its not working.
    I am using PCF SCDF tile.