Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    ehausig
    @ehausig
    Hi, I'm new to CloudFlow and have been working through the tutorial and am trying to reconcile the build/deploy pipeline with our environment:
    6 replies
    dlmutart
    @dlmutart
    Looks like the Cloudflow operator version 2.0.10 has a dependency on a vulnerability fix using a skuber version that is no longer available in Maven repo. We get an error: [ERROR] Failed to execute goal on project approval: Could not resolve dependencies for project artifact:some:jar:1: Could not find artifact io.skuber:skuber_2.12:jar:2.4.1-cve-fix-a8d7617c in java. I noticed a note in the 2.0.10 build file about // skuber version 2.4.0 depends on akka-http 10.1.9 : hence overriding with akka-http 10.1.12 to use akka 2.6 remove this override once skuber is updated so wondering if pull request #628 is intended to fix this?
    3 replies
    Thomas
    @thomasschoeftner
    Hi - quick question:
    Is it safe to upgrade from Cloudflow 2.0.7 to 2.0.10 directly?
    10 replies
    shivgit87
    @shivgit87
    Unable to Produce data to External Kafka using CloudFlow and Flink (Java), But with the same Code I am able to successfully Produce to Kafka when I use only Flink. Can anyone help here ? I am using CloudFlow 2.0.9 version.
    3 replies
    Thomas
    @thomasschoeftner
    Hi, about the Cloudflow container images / PODs:
    How can I properly attach a sidecar container in the PODs running AkkaStreamlets?
    I haven't found anything revealing in the docs and the sources...
    In our case a lot of system integration with our eco-system happens via sidecars.
    Thanks for any pointers!
    33 replies
    DarthKrab
    @DarthKrab

    Hi. I wrote earlier about this problem. When I recreate the jobmanager container of a streamlet in my pipeline (or it crashes for some reason), after starting, I see errors like this in the JobManager's logs:
    2020-09-22T08:51:55.311Z ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [{}] - Exception occurred in REST handler: Job ed952687752d2a5b2c60d843d7e5605f8 not found
    2020-09-22T08:52:25.485Z ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [{}] - Exception occurred in REST handler: Job ed952687752d2a5b2c60d843d7e5605f8 not found
    2020-09-22T08:52:55.673Z ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [{}] - Exception occurred in REST handler: Job ed952687752d2a5b2c60d843d7e5605f8 not found
    ....

    The directory with checkpoints exists and contains binary data and metadata: http://joxi.ru/LmGPZgeTlvj9G2
    But the jobmanager doesn't find it.
    Тhe streamlet has the following settings:
    http://joxi.ru/eAOPkEYTkdegbr

    In a prod environment we can't just redeploy the pipeline, because the data is important.

    It is necessary that the streamlet after re-creation continues to work from the moment of the fall. Maybe someone has already encountered this problem. What settings should I check?

    16 replies
    shivgit87
    @shivgit87
    We are seeing failures in deployment for our Cloudflow application , Is it mandatory to have cloudflow and AKS to be on same version or is it ok to be on different version (As in our case CloudFlow is on 2.0.9 and AKS is of 2.0.10 version).
    1 reply
    Timothy Zhang
    @echang0929
    Hi team, why did i get version 2.0.8 always even though I install cloudflow with image version 2.0.10?
    ➜ kubectl cloudflow version
    2.0.8 (645-df682e0)
    5 replies
    Here is the description of the operator pod:
    Containers:
    cloudflow-operator:
    Container ID: containerd://eb51669dad9c723c2ccab2f5ecd866e7d24d8d9a22292221431fdbecf96b6fc2
    Image: lightbend/cloudflow-operator:2.0.10
    Image ID: docker.io/lightbend/cloudflow-operator@sha256:14e86e10090f371b7ae98907d5eb5ded220a821a4da9ca169ea80674758bc10d
    Port: 5001/TCP
    Host Port: 0/TCP
    State: Running
    Started: Sun, 27 Sep 2020 02:34:11 +0800
    DarthKrab
    @DarthKrab
    Hi. Нow should I install cloudflow in an openshift cluster if I want to use an external kafka? Bootstrap-install-script is very convenient, but it installs Strimzi by default.
    2 replies
    Thomas
    @thomasschoeftner
    Hi, checking the current Cloudflow sources, I can see how to define the number of partitions per topic.
    Also, the replicationfactor seems to be handled in TopicActions.createAction.
    Is there also a way to define the retention time per topic from Cloudflow config?
    On our cluster, data is piling up fast and IMO 1d retention time should do...
    26 replies
    DarthKrab
    @DarthKrab

    Hi. I need help)
    I am using flink. I'm trying to parameterize the configuration of my pipeline, using environment variables.
    Тhere is an example here https://cloudflow.io/docs/dev/develop/cloudflow-configuration.html :
    ...
    config-parameters {
    // config parameter values go here
    my-config-parameter = "some-value"
    another-config-parameter = "default-value"
    another-config-parameter = ${?MY_VAR}
    }
    ....

    In my case, I have two files: app.conf and env.conf

    The app.conf file has lines:

    config-parameters{
    output-sink-type = "kafka",
    kafka-bootstrap-servers = "cloudflow-kafka-brokers.cloudflow.svc:9092",
    kafka-topic = "topic-name${TOPIC_SUFFIX}",
    transaction-timeout-ms = "900000"
    flink-checkpoint-timeout = 1 second,
    }

    The env. conf flie has lines:

    cloudflow.runtimes.flink {
    kubernetes.pods {
    job-manager.containers.container {
    env = [
    { name = "TOPIC_SUFFIX"
    value = "-dev"
    }
    ]
    resources {
    requests {
    cpu = "500m"
    memory = "2Gi"
    }
    limits {
    memory = "2Gi"
    }
    }
    }

    task-manager.containers.container {
      env = [
        { name = "TOPIC_SUFFIX"
          value = "-dev"
        }
      ]
      resources {
        requests {
          cpu = "1"
          memory = "4Gi"
        }
        limits {
          memory = "6Gi"
        }
      }
    }

    }
    }

    I use them in deployment: kubectl-cloudflow deploy /path/pipe.json --conf app.conf --conf env.conf

    After that, I see an error in the flink logs:

    2020-10-02 14:40:32
    org.apache.kafka.common.errors.InvalidTopicException: Topic 'file-checksum-check-success-bp${DEV_TOPIC_SUFFIX}' is invalid

    Why isn't the change inserted in the topic title?

    7 replies
    orElse
    @orElse
    Hi,
    A quick question regarding renaming a streamlet name in the blueprint - when doing this and redeploying the old deployment was kept alive - should the cloudflow operator take care of this - or is it intended as a manual step to delete the old deployment of the streamlet manually?
    4 replies
    kozy
    @kozy

    Good evening everyone,

    on startup I would like to have a peek at the last message in the topic to compare it and see if I missed any. I need to do this because my source does not wait for me and I will have to fetch and prepend any missed messages.
    The approach I came up with is to use a SourceWithContext and commit the previous message each time. This way there will always be an ucommited last message replayed which I can compare. Does this sound reasonable? Or is there a better way? Thank you!

    4 replies
    DarthKrab
    @DarthKrab
    Hello. Does version cloudflow 2.0.5 support using external Kafka?
    3 replies
    Alex Sergeenko
    @Lockdain
    Hi!
    I've noticed that a kind of streamlets ASCII graph is printed to the console during a local pipeline build (at least for 2.0.5). That feature looks pretty handy so I have an idea to save the visualization as a sort of pipeline documentation. Can anybody point me to the best solution?
    7 replies
    shivgit87
    @shivgit87
    I am getting following error , when I am trying to deploy my streamlet application :-
    [IJ]sbt:cf-data-platform-data-pipeline> last data-pipeline-blueprint / runLocal
    [error] java.util.NoSuchElementException: None.get
    [error] at scala.None$.get(Option.scala:529)
    [error] at scala.None$.get(Option.scala:527)
    [error] at cloudflow.sbt.CloudflowLocalRunnerPlugin$.$anonfun$findLogLibsInPluginClasspath$2(CloudflowLocalRunnerPlugin.scala:183)
    [error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
    [error] at scala.collection.immutable.List.foreach(List.scala:392)
    [error] at scala.collection.TraversableLike.map(TraversableLike.scala:238)
    [error] at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
    [error] at scala.collection.immutable.List.map(List.scala:298)
    [error] at cloudflow.sbt.CloudflowLocalRunnerPlugin$.findLogLibsInPluginClasspath(CloudflowLocalRunnerPlugin.scala:183)
    [error] at cloudflow.sbt.CloudflowLocalRunnerPlugin$.$anonfun$projectSettings$11(CloudflowLocalRunnerPlugin.scala:102)
    [error] at cloudflow.sbt.CloudflowLocalRunnerPlugin$.$anonfun$projectSettings$11$adapted(CloudflowLocalRunnerPlugin.scala:88)
    [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
    [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
    [error] at sbt.std.Transform$$anon$4.work(Transform.scala:67)
    [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
    [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
    [error] at sbt.Execute.work(Execute.scala:290)
    [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
    [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
    [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
    [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    [error] at java.base/java.lang.Thread.run(Thread.java:834)
    [error] (data-pipeline-blueprint / runLocal) java.util.NoSuchElementException: None.get
    [IJ]sbt:cf-data-platform-data-pipeline>
    shivgit87
    @shivgit87
    No one here has encountered above issue?
    6 replies
    orElse
    @orElse
    Hi,
    a question regarding the experimental clustering feature (https://cloudflow.io/docs/current/develop/clustering-akka-streamlet.html) - on which level will the cluster be formed - do only the same streamlet form one? or will all streamlets of the service/namespace form one?
    11 replies
    Bilna P
    @Bilna
    Hi, does flink 1.10 supports azure data lake gen2? if you have any example to share, please provide the link..
    1 reply
    Raymond Roestenburg
    @RayRoestenburg
    FYI Cloudflow 2.0.11 has been released: https://github.com/lightbend/cloudflow/releases/tag/v2.0.11
    orElse
    @orElse
    Hi it is me again with another question - I saw that the streamlets are started with the jmx prometheus agent - what would be the easiest way to let it also scrape my own custom metrics? Use a volume mount to override the default prometheus.yaml?
    5 replies
    DarthKrab
    @DarthKrab
    Hi. Are there plans to integrate cloudflow and istio? It might help to carefully trace the problems of interaction between the components of the platform.
    4 replies
    Evgeniy
    @jenoOvchi
    Hello!
    In the cloudflow 2.0.5 Flink Operator by default run Prometheus agent on port 2050 (https://github.com/lightbend/cloudflow/blob/v2.0.5/external/multi-base-images/flink/flink-entrypoint.sh), but lyft/flinkk8soperator v.0.4.0, that used in Cloudflow 2.0.5, use port 50101 for MetricsQueryDefaultPort (https://github.com/lyft/flinkk8soperator/blob/v0.4.0/pkg/controller/flink/config.go) and do not expose port 2050. What is the easiest way to expose Flink metrics after Cloudflow Application was deployed?
    29 replies
    dlmutart
    @dlmutart
    Looks like Cloudflow Events 2.0.11 isn't available in Mvn Repository, only RC1. We're still dealing with the bad version dependency on skuber in 2.0.10. Any idea when that Cloudflow Events RC1 will be stable?
    1 reply
    Serjkea
    @Serjkea
    This message was deleted
    3 replies
    Serjkea
    @Serjkea
    Hello! I have problem when I start my test project with "runLocal" command: [error] java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7
    [error] at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45)
    [error] at kafka.api.ApiVersion.compare(ApiVersion.scala:139)
    [error] at kafka.api.ApiVersion.compare$(ApiVersion.scala:138)
    [error] at kafka.api.KAFKA_2_5_IV0$.compare(ApiVersion.scala:339)
    [error] at kafka.api.KAFKA_2_5_IV0$.compare(ApiVersion.scala:339)
    [error] at scala.math.Ordered.$greater$eq(Ordered.scala:91)
    [error] at scala.math.Ordered.$greater$eq$(Ordered.scala:91)
    [error] at kafka.api.KAFKA_2_5_IV0$.$greater$eq(ApiVersion.scala:339)
    [error] at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1529)
    [error] at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1237)
    [error] at net.manub.embeddedkafka.ops.KafkaOps.startKafka(kafkaOps.scala:52)
    [error] at net.manub.embeddedkafka.ops.KafkaOps.startKafka$(kafkaOps.scala:26)
    [error] at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:52)
    [error] at net.manub.embeddedkafka.ops.KafkaOps.startKafka(kafkaOps.scala:65)
    [error] at net.manub.embeddedkafka.ops.KafkaOps.startKafka$(kafkaOps.scala:57)
    [error] at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:52)
    [error] at net.manub.embeddedkafka.ops.RunningKafkaOps.startKafka(kafkaOps.scala:89)
    [error] at net.manub.embeddedkafka.ops.RunningKafkaOps.startKafka$(kafkaOps.scala:86)
    [error] at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:52)
    [error] at net.manub.embeddedkafka.EmbeddedKafka$.start(EmbeddedKafka.scala:70)
    [error] at cloudflow.sbt.CloudflowLocalRunnerPlugin$.setupKafka(CloudflowLocalRunnerPlugin.scala:202)
    [error] at cloudflow.sbt.CloudflowLocalRunnerPlugin$.$anonfun$projectSettings$11(CloudflowLocalRunnerPlugin.scala:128)
    [error] at cloudflow.sbt.CloudflowLocalRunnerPlugin$.$anonfun$projectSettings$11$adapted(CloudflowLocalRunnerPlugin.scala:88)
    [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
    [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
    [error] at sbt.std.Transform$$anon$4.work(Transform.scala:68)
    [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
    [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23) [error] at sbt.Execute.work(Execute.scala:291)
    [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
    [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
    [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
    [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    [error] at java.base/java.lang.Thread.run(Thread.java:834) Try with Scala 2.12.10-2.12.12...
    10 replies
    Thomas
    @thomasschoeftner

    Hi - anyone else have a problem upgrading from 2.0.10 to 2.0.11?
    I created an issue for this - lightbend/cloudflow#809
    In our case, the operator enters a Crashloop complaining about missing double-quotes on a config value...

    I should add: I did not undeploy/redeploy the running streamlets - they're still from the 2.0.10 deployment.

    6 replies
    DarthKrab
    @DarthKrab
    Hi. I need to set the value of the taskSlots parameter for flink taskmanager. This parameter is present in flinkapplication crd (https://github.com/lyft/flinkk8soperator/blob/master/docs/crd.md) How can this be done using The Cloudflow Configuration Model?
    5 replies
    Max Shcherbakov
    @MaxSbk
    Hello! I have problem with using akka-stream-alpakka-s3 at Cloudflow 2.0.5. Please tell me how and where to describe the settings for the alpakka.s3 block? Using the example of local.conf, I tried to specify this block at cloudflow.runtimes, streamlets.<some_streamlet>.config and even at the root of the file local.conf, but the result is negative
    5 replies
    orElse
    @orElse

    Hi,
    Is there a way to hook into a streamlet (pod) being stopped gracefully (e.g. due to a deployment when the old pod is stopped)
    I was trying to hook into the life cycle of the actor system with:

    CoordinatedShutdown(system).addTask(CoordinatedShutdown.PhaseBeforeActorSystemTerminate, "ShutdownTask") { () =>
          log.info("Stopping .....  in CoordinatedShutdown")
          Future.successful(Done)
        }

    or

    system.registerOnTermination(shutdown())

    but this did not seem to work

    12 replies
    Thomas
    @thomasschoeftner

    Would it make sense to allow including other hocon configs (via include) in blueprint.conf?
    We have a number of config properties which are independent from the deployment environment, but explicitly need to be set in every blueprint.
    Right now, this boils down to copy/pasting an extra config section in each blueprint.conf and referencing that from the topics section.

    Extracting these common settings into a dedicated conf file and deploying that via kubectl cloudflow would not really help here, because blueprint.conf cannot refer to yet-unknown variables. We'd have to reconfigure every single topic in that config file...

    35 replies
    Evgeniy
    @jenoOvchi

    Hi!
    If i delete Flink JobManager pod or it restarts with some other reasons than Flink cluster goes broken:

    • no jobs in Flink UI;
    • error in JobManager:
      2020-09-17T10:47:24.582Z ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [{}] - Exception occurred in REST handler: Job c18eebfa9734c6b007373ae5fd7dd65e not found
      2020-09-17T10:47:43.614Z ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [{}] - Exception occurred in REST handler: Job c18eebfa9734c6b007373ae5fd7dd65e not found
    • error in TaskManager:
      2020-09-17T10:46:40.776Z WARN akka.remote.ReliableDeliverySupervisor [{akkaSource=akka.tcp://flink@10.129.117.209:35258/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fflink%40credit-history-pipe-rls-cluster-updater-d7af0c1c%3A6123-0, akkaTimestamp=10:46:40.769UTC, sourceActorSystem=flink, sourceThread=flink-akka.remote.default-remote-dispatcher-6}] - Association with remote system [akka.tcp://flink@credit-history-pipe-rls-cluster-updater-d7af0c1c:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@credit-history-pipe-rls-cluster-updater-d7af0c1c:6123]] Caused by: [java.net.ConnectException: Connection refused: credit-history-pipe-rls-cluster-updater-d7af0c1c/172.30.54.198:6123]
      2020-09-17T10:46:50.793Z WARN akka.remote.transport.netty.NettyTransport [{akkaSource=NettyTransport(akka://flink), akkaTimestamp=10:46:50.792UTC, sourceActorSystem=flink, sourceThread=New I/O boss #3}] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: credit-history-pipe-rls-cluster-updater-d7af0c1c/172.30.54.198:6123

    Is it actual for someone else (for Flink Streamlets) or it's only our problem? Maybe we doing something wrong?

    18 replies
    Max Shcherbakov
    @MaxSbk
    Hi!
    Is there a way to manually start / restart a Flink based streamlet from a savepoint with a state, by analogy with a regular flink job (./bin/flink run -s <savepointPath> ...)?
    4 replies
    Thomas
    @thomasschoeftner

    Hi, our Cloudflow (2.0.10) application currently gives us some troubles with unwanted message re-delivery from Kafka topics (via inlets).

    We use mapConcat to produce many messages (with the same kafka offset) from a single inbound request.
    As far as I can see, all messages get processed and committed properly but when the streamlet gets restarted by GKE for some reason (e.g. if pod gets relocated), we can see a lot of re-processing of already handled messages.

    I would be great if a Cloudflow pro had time to look at these questions:
    (1) Cloudflow does not commit an offset to Kafka unless it has received another message, right?
    In more detail:
    (2) Cloudflow tracks the committable-offsets per Kafka-partition. - I presume that an offset will only get committed if another message has received from that same partition, right? - Not from another partition on that topic.
    (3) Will Cloudflow commit processed and yet uncommitted offsets when the Streamlet pod is shut down? - Usually GKE will allow for 30s grace time, iirc.
    (4) Is it possible that there is a little SNAFU on Cloudflow/alpakka-kafka shutdown timeouts vs. K8S shutdown grace time - so that some messages my not get commited?

    31 replies
    Raymond Roestenburg
    @RayRoestenburg
    Hi @/all we're moving to zulip: https://cloudflow.zulipchat.com/ please signup! Zulip has a lot of great features and is a great improvement compared to gitter. See you there!
    Please post any new questions there, we'll deprecate the gitter channel soon..
    7 replies
    Raymond Roestenburg
    @RayRoestenburg
    @/all I'm going to try and archive gitter and import the messages into zulip. Please post new messages to Zulip instead of here.
    1 reply
    cloudflow.zulipchat.com
    Raymond Roestenburg
    @RayRoestenburg
    (It looks like it's only possible for a new organization though)
    In any case, please post new messages in zulip, and if you have unanswered questions, it would be great if you could repost those on zulip.
    Raymond Roestenburg
    @RayRoestenburg
    From today we're moving officially off Gitter, and onto Zulip. :tada:
    The Lightbend Cloudflow team will only be answering questions on the new channel https://cloudflow.zulipchat.com/ from today. Thanks Gitter for the ride so long!
    Rohit Kumar
    @rohitsrmuniv
    Is there any date to cloudflow release 2.023?
    Sven Ludwig
    @iosven
    hi, I am not sure if this is the right place to ask, but our enterprise is now reaching the point where we need to support MQTT 5 (as a follow-up to MQTT 3.1.1). I remember that alpakka-mqtt streaming does not yet support MQTT 5. But maybe the Eclipse Paho based connector does, so the question is: Does alpakka-mqtt-streaming https://doc.akka.io/docs/alpakka/current/mqtt.html support MQTT 5 or is the support soon coming?
    Background of this question is also that I'd like us to avoid moving to an entirely different technology stack in that backend service just because we need to support MQTT 5.
    orElse
    @orElse
    @rohitsrmuniv & @iosven -> conversations were moved to zulip https://cloudflow.zulipchat.com/ - this channel is more or less inactive