Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Shehan Abayagunawardena
    @shehanab
    I have integrated Keycloak for authentication and authorization. I manage to login into the system successfully every time I try to login to the app using http://localhost:9393/dashboard. But when I logout and login it takes me to http://localhost:9393/ which loads just a json object. Hence I have to load the URL manually. Find more details here spring-cloud/spring-cloud-dataflow#4470 @sabbyanandan @jvalkeal
    2 replies
    kumarprashant2103
    @kumarprashant2103

    Hi,
    We recently installed SCDF on our Kubernetes cluster. We can load the docker hub applications: docker:springcloudstream/cassandra-sink-kafka:3.0.1 and executed the task successfully.
    We created our own application and created an image in our Docker repository. When we try to execute the task with our own application , it is giving error : Invalid repository name: WCC2.0-batchinitialization-4-46176.
    Do we need to specify our Docker details specifically at any of the configuration file ? Here is the complete error log from the task execution :

    io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/kub-wcc-pdev/pods/initialization-batch-o86jqw9j0x/log?pretty=false&tailLines=500. Message: container "initialization-batch-1k7yj5x34k" in pod "initialization-batch-o86jqw9j0x" is waiting to start: InvalidImageName. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=container "initialization-batch-1k7yj5x34k" in pod "initialization-batch-o86jqw9j0x" is waiting to start: InvalidImageName, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={}).

    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)

    2021-04-08 14:31:07.569 WARN 1 --- [p-nio-80-exec-1] ApplicationConfigurationMetadataResolver : Failed to retrieve properties for resource Docker Resource [docker:pdev/WCC2.0-batchinitialization-4-46176] because of IllegalArgumentException: Invalid repository name: WCC2.0-batchinitialization-4-46176

    2021-04-08 14:31:07.570 WARN 1 --- [p-nio-80-exec-1] ApplicationConfigurationMetadataResolver : Failed to retrieve properties for resource Docker Resource [docker:pdev/WCC2.0-batchinitialization-4-46176] because of IllegalArgumentException: Invalid repository name: WCC2.0-batchinitialization-4-46176

    2021-04-08 14:31:07.635 INFO 1 --- [p-nio-80-exec-1] o.s.c.d.s.k.KubernetesTaskLauncher : Preparing to run a container from Docker Resource [docker:pdev/WCC2.0-batchinitialization-4-46176]. This may take some time if the image must be downloaded from a remote container registry.

    2021-04-08 14:31:07.636 INFO 1 --- [p-nio-80-exec-1] o.s.c.d.s.k.DefaultContainerFactory : Using Docker image: pdev/WCC2.0-batchinitialization-4-46176

    2021-04-08 14:31:07.636 INFO 1 --- [p-nio-80-exec-1] o.s.c.d.s.k.DefaultContainerFactory : Using Docker entry point style: exec

    kumarprashant2103
    @kumarprashant2103
    @sabbyanandan - We set up this parameter to pull the images from our docker registry :
    • name: SPRING_CLOUD_DEPLOYER_KUBERNETES_IMAGE_PULL_SECRET
          valueFrom:
            secretKeyRef:
              key: .dockerconfigjson
              name: regcred
      This is getting resolve inside the POD # echo $SPRING_CLOUD_DEPLOYER_KUBERNETES_IMAGE_PULL_SECRET
      {"auths":{"https://<SERVER_NAME>/":{"username":"<user_id>","password":"<password>","email":"<email_id>","auth":"3BhcOTA0Mw=="}}}
    but we are still getting this error : 2021-04-09 19:15:22.435 WARN 1 --- [-nio-80-exec-10] ApplicationConfigurationMetadataResolver : Failed to retrieve properties for resource Docker Resource [docker:pdev/pdev :-interactionref-382-46196] because of HttpClientErrorException.Unauthorized: 401 Unauthorized: [{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"pdev/pdev","Action":"pull"}]}]}
    Sathish Kumar
    @sat_cse28_twitter
    Hi @sabbyanandan is there a way we can deploy our application in SCDF with internal loadbalancer from cloudprovider instead of using deployer.app.kubernetes.createLoadBalancer=true on Kubernetes
    3 replies
    sakinon
    @sakinon
    image.png

    Hi all,I'm unable to launch source and sink streaming apps within minikube k8s. Instruction used here : https://dataflow.spring.io/docs/installation/kubernetes/kubectl

    Error in the pods :

    Failed to apply default image tag "//root/source-0.0.1.jar": couldn't parse image reference "//root/source-0.0.1.jar": invalid reference format

    3 replies
    sakinon
    @sakinon
    I am using minikube on my local machine and here is the spring-cloud-dataflow-server/docker-compose.yml contents
    2 replies
    image.png
    kumarprashant2103
    @kumarprashant2103
    How do we pass the certificate file in the deployment yaml. We are getting the below error when we try to execute our applications from private docker registry : Failed to retrieve properties for resource Docker Resource [docker:dtrprod.kbm1.loc/pdev/pdev:test-interactionref-382-46196] because of SunCertPathBuilderException: unable to find valid certification path to requested target
    6 replies
    MarsGradvius
    @MarsGradvius
    I am currently trying to use the composer task runner to run a serues lf batch tasks. It all woros fune however recently i added uaa auth to the site and now i am getting 401 unauthorized messages from the CTR app.
    I saw some documentation ssying i could specify a username and password for basic auth but that doesnt seem
    to work for oath, does anyone have any experience with this?
    David Bahat
    @dbahatSAP
    Hi @sabbyanandan.
    Are there any plans to release a new SCDF version anytime soon?
    Releases seemed to stop with 2.7.1 a few months ago, and there are some important fixes in both SCDF server it's upstream dependencies (e.g. spring-boot) released since January
    1 reply
    sakinon
    @sakinon
    image.png
    Hi All, I have my custom source and sink applications written with java-scdf.I am using kafka as messaging middleware. So my question is : "-- are those sink instances consume same messages or are they consuming same kafka topic and sharing the data from it for the below pipeline design"
    10 replies
    My purpose was using that destination component as a load balancer, So does it work for that ? If not is there a way to do that by giving some configuration ?
    Venkob
    @Venkob
    Hi All, I have my custom source, processor and sink applications written with java .I am using kafka as messaging middleware. I have the registered all three apps and successfully Deployed in SCDF.s o my question is : "-- How to schedule a Stream job to run Daily.
    Basically am reading csv file and dumping to Database. file May Come any Time.
    In SCDF Automatically Stream Get executed like daily or weekly bases. Is there any scheduler for Streams In SCDF?
    1 reply
    jimKarsh
    @jimKarsh
    Hello! I'm currently pitching introducing SCDF at our organization. I believe it fits our functional requirements quite well for both migration from monolith to event-driven microservices as well as the vast majority of our Integration needs currently running exclusively on batch. Anyone here who has undergone that journey of initial implementation + scaling the processes willing to share personal experiences with it would be highly appreciated (I've, so far, been met with an equal amount of skepticism and enthusiasm). Also if this is not the right place to seek this info...sorry! (and please a gentle nudge to the right place for such a request :))
    1 reply
    Venkob
    @Venkob
    Hi All , I am running SCDF Server with the below command , java -jar spring-cloud-dataflow-server-2.7.1.jar --spring.datasource.url=jdbc:mysql:database --spring.datasource.username=username --spring.datasource.password=password --spring.datasource.driver-class-name=org.mariadb.jdbc.Driver &, Problem here is even though mentioning Mysql Credential while running it is connect default to H2 data base which i dont want, I want to connect with Mysql DB and want to register and launch Spring cloud task App in SCDF. Any recommendation plz.
    3 replies
    Nicolas Schwarzentrub
    @niggusch

    Hi there, we are running SCDF on Openshift with entryPointStyle=boot in openshift. We have some Tasks with metadata/properties. If we create the task from within the ui and fill the parameter through the options dialog and start it, everything works fine, the parameters are correctly mapped and put into env SPRING_APPLICATION_JSON. But if we create the same task without any options and pass the parameters as arguments things seem to get messeed. The Parameters are passed to the container as args instead of be set to the SPRING_APPLICATION_JSON and it seems that its implementation only evaluates the SPRING_APPLICATION_JSON and therefore fails, since the passed arguments have not effect.
    Excerp of the pod yaml:

    spec:
        containers:
            - args:
                - '--myParam=value
                - '--spring.cloud.data.flow.platformname=default'
                - '--spring.cloud.task.executionid=610'
              env:
                - name: SPRING_CLOUD_CONFIG_URI
                  value: 'http://config-server:8080'
                - name: PGPASSFILE
                  value: /var/lib/pgsql/.pgpass
                - name: SPRING_APPLICATION_JSON
                  value: >-
                    {"spring.datasource.username":"**","spring.cloud.task.name":"replay-vmnr","spring.datasource.url":"jdbc:postgresql://postgresql:5432/**?currentSchema=batch","spring.datasource.driverClassName":"org.postgresql.Driver","spring.datasource.password":"***"}

    Any suggestions on that?

    LukaszKusmierczyk
    @LukaszKusmierczyk

    Hi all,
    When I deploy my stream in SCDF, exchanges are created in rabbitmq with names like step1step2step3-in-0 and step1step2step3-out-0.

    This is how I create stream:
    stream create --name myapp-init-v1 --definition ":myapp-init-v1.input > comm-myapp > :myapp-init-v1.output"

    comm-myapp is a processor with stream definition step1|step2|step3

    I expect to have myapp-init-v1.input and myapp-init-v1.output exchanges in rabbitmq.
    I use SCDF 2.7.1 with rabbitmq 3.6.0.
    Could you help me?

    Venkob
    @Venkob
    Hi all, I am able to connect SCDF with Mysql 8 using maria db driver ,but problem is in my app i have used in my app mysql jdbc driver
    spring:
    datasource:
    driverClassName: 'com.mysql.cj.jdbc.Driver'
    url: 'jdbc:mysql://localhost:3306/dbname'
    username: 'uname'
    password: 'password' , i am getting problem is here java.lang.IllegalStateException: Cannot load driver class: org.mariadb.jdbc.Driver, plz help me out to work app with mysql
    1 reply
    Mansingh Shitole
    @msshitole

    Stream and application monitoring is not showing up in the latest SCDF v2.7.1 (spinned up using docker-compose and prometheus-docker)
    It was used to work in previous SCDF and spring cloud versions.
    Here are the details:

    • configs added in the application.properties
      management.endpoint.prometheus.enabled=true
      management.endpoints.web.exposure.include=app,health,info,prometheus
    • dependencies added in the pom.xml
      <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
          <groupId>io.micrometer</groupId>
          <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <!-- Rsocket related libs -->
      <dependency>
          <groupId>io.micrometer.prometheus</groupId>
          <artifactId>prometheus-rsocket-spring</artifactId>
          <version>1.3.0</version>
        </dependency>
        <dependency>
          <groupId>io.rsocket</groupId>
          <artifactId>rsocket-core</artifactId>
          <version>1.0.2</version>
        </dependency>

      <!-- micrometer common libs -->

      <dependency>
          <groupId>org.springframework.cloud.stream.app</groupId>
          <artifactId>app-starters-micrometer-common</artifactId>
          <version>2.1.6.RELEASE</version>
          <exclusions>
            <exclusion>
              <artifactId>spring-security-config</artifactId>
              <groupId>org.springframework.security</groupId>
            </exclusion>
            <exclusion>
              <artifactId>spring-cloud-services-starter-config-client</artifactId>
              <groupId>io.pivotal.spring.cloud</groupId>
            </exclusion>
            <exclusion>
              <artifactId>*</artifactId>
              <groupId>org.springframework.boot</groupId>
            </exclusion>
            <exclusion>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-stream</artifactId>
            </exclusion>
          </exclusions>
        </dependency>
      Here is the sample application:

      https://github.com/msshitole/demo-source

    P.S

    <spring-cloud.version>2020.0.2</spring-cloud.version>
    spring-boot latest v2.4.5

    @sabbyanandan Could you please check and assist here. Thank you in anticipation!

    5 replies
    gpehh
    @gpehh
    When I run SCDF composed task running on oracle database, I get "Cannot load driver class: oracle.jdbc.OracleDriver" error. can you help me
    5 replies
    Venkob
    @Venkob
    Hi all, In SCDF how to change root directory path, by default all logs and etc is writing and from C drive of windows, here i want to change C to D drive or other.
    1 reply
    dcastracucchi
    @dcastracucchi
    Hi all , im new in dataflow , im ha installed it on k8 but i receive this error for the strem:
    Failed to apply default image tag "//org.springframework.cloud.stream.app:s3-source-rabbit:jar:3.0.1": couldn't parse image reference "//org.springframework.cloud.stream.app:s3-source-rabbit:jar:3.0.1": invalid reference format
    someone can help me?
    9 replies
    thanks
    Vladimir Avdoshka
    @vavdoshka
    Hello, has been anyone practicing GitOps approach to stream deliveries and updates in K8s? As I understand there are two ways how one can do this - both through k8s jobs (with some helm hooks potentially) with either an SCDF shell being fed with a list of commands or some custom implementation that uses REST API and custom DSL or something like this. In that case, either a list of commands or a custom DSL structure would be under source control. Is there any other approach, can you share your thoughts on this topic, please?
    7 replies
    gpehh
    @gpehh
    Hi all, SCDF is running on oracle but I get this error “Cannot load driver class: oracle.jdbc.OracleDriver” while running Composed Task Runner.CTR can run in a local environment with which annotation. Like @Enabledataflowserver, @EnableSkipperServer.This way I can customize it with oracle driver.
    Anupam Behera
    @thekindler
    How do i configure scdf apps to send messages to the kafka topic created by dataflow, i.e the topics with streamname.appname and not to methodname-out-0? .. @sabbyanandan
    2 replies
    zhaodongru
    @zhaodongru
    Hi, I have deployed SCDF 2.7.0 to k8s. When I'm trying to deploy a stream i got the following exception java.lang.IllegalArgumentException: No platforms configured.
    LukaszKusmierczyk
    @LukaszKusmierczyk
    Hello guys,
    Maybe stupid question, but is there a possibility to use the same deplyed instance of application in two different streams ? As I understand when I create and deploy 2 streams that uses let say prebuilt http app I will end up with 2 separate instances of http on my environment.
    natraj09
    @natraj09
    Hello , is there any update to this spring-cloud/spring-cloud-dataflow#4290.
    1 reply
    Anupam Behera
    @thekindler
    Hi everyone, the service names generate by dataflow for the deployed streams are of the format:, stream-name-appname-<some version no>. The version no keeps changing after every undeploy/deploy option. Is it possible to change the name of this service name to something fixed, so that it can be easily ,managed. @sabbyanandan
    2 replies
    Jeon ChangMin
    @rafael81
    Hi everyone, If i should operate so many stream , Should I deploy many skipper?
    4 replies
    Jordan Finci
    @finci-square
    Hi there, high level conceptual question: We intend to build a large scale batch implementation with many worker nodes that run independent job initiated via a web requests. Spring Batch documentation is clear on how to do this. What seems to remain a mystery is whether Spring Cloud Dataflow can provide cluster orchestration (failure detection, failover, etc) for the cluster and launch jobs as a result of web requests. is that possible with dataflow? Any supporting documentation?
    Anupam Behera
    @thekindler
    @sabbyanandan , i am continously getting unauthorised issue while accessing the rest apis of scdf. I have configured scdf with keycloak auth and have deployed in k8s. I am able to successfully login in the ui. @sabbyanandan any suggestion how to debug this. Can i enable fulls logs in scdf server?
    1 reply
    Anupam Behera
    @thekindler
    is there any configuration to allow embedding of scdf ui in iframe
    HaoTanAtMLB
    @HaoTanAtMLB
    Hi is there a way to clean up a stream when Skipper has lost sync of the state of the stream? Dataflow server shows the stream is undeployed, all pods related to the stream are still running, but skipper/dataflow server does not allow for me to delete the stream... any help would be appreciated.
    Jeon ChangMin
    @rafael81
    Hello, I created stream between "http" and "log" in kubernetes using minikube.
    The each container is being launching separately. But the each container's image name is invalid like below //org.springframework.cloud.stream.app:http-source-kafka:jar:3.0.1.
    2 replies
    image.png
    Jeon ChangMin
    @rafael81
    image.png
    Mike
    @patelm5
    Hi. Do you think it's possible as part of the 2.8.0 release of scdf that the local docker-compose solution has broken ? When I run docker compose up with DATAFLOW_VERSION=2.8.0, I can't deploy jvm 11 apps ( I get an unsupported class version error stating that server expects < version 52 ). Works fine with DATAFLOW_VERSION=2.7.2.
    Mike
    @patelm5
    Oh, nvm, I think actually it's because of the spring-cloud-skipper-server version. Needs to be 2.7.0-jdk11