Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Erik Salter
    @an1310
    Hi there -- quick question about the Java Client SDK across multiple microservice instances. If I register the microservice's client to listen for events -- client.twin().registerForFeaturesChanges("globalFeaturesHandler", change -> ... - will all instances of the microservice receive these events?
    Thomas Jaeckle
    @thjaeckle

    hi @an1310
    currently, the Websocket (which the Things Java client connects to) does a classical pub/sub, so every client running the same code would get all changes
    we were thinking about whether it would make sense to (optionally) e.g. provide a common "group-identifier" so that only one of the client sessions would get an event, load balancing
    is this what you had in mind? we're still not convinced if this is needed

    because the alternative would be to create a managed connection and e.g. use a "Http" connection (https://www.eclipse.org/ditto/connectivity-protocol-bindings-http.html) in order to publish event
    a loadbalancer before the endpoint where the events are published to could then do the load balancing

    Erik Salter
    @an1310
    Hi @thjaeckle -- yes, load balancing is what I had in mind.
    Erik Salter
    @an1310
    @thjaeckle Is this what you were referring to with group identifiers. eclipse/ditto#878
    Thomas Jaeckle
    @thjaeckle
    @an1310 yes and no. This just merged PR is the foundation to do that in websockets as well, but we did not yet add it to the Websocket api. If you could create a new issue for that, it would be great (from a use case perspective).
    mfbatra
    @mfbatra

    Hello Everyone,

    Just a general Question - Does Ditto offer Event Handling? It has been great uptill now.

    Best,

    Thomas Jaeckle
    @thjaeckle
    @mfbatra what exactly do you mean with "Event Handling"?
    Thomas Jaeckle
    @thjaeckle
    @/all we released Eclipse Ditto 1.5.0 yesterday: https://www.eclipse.org/ditto/2020-12-10-release-announcement-150.html
    mfbatra
    @mfbatra
    @thjaeckle For example, I know that my IOT device should not reach a value after a specific threshold and I want to get a warning when that happens and also perform some prevention mechanism like sending an event to my Physical twin to stop.
    Thomas Jaeckle
    @thjaeckle
    @mfbatra you can use Connections (e.g. a HTTP connection publishing such notifications to an HTTP endpoint of your choice) in order to do that
    you can also define RQL "filters" in order to define such thersholds, e.g. gt(features/temperature/properties/value,20) to only invoke publishing a notification under a certain condition
    you can also use the WebSocket to get such notifications, but then you'll have to maintain an opened WS connection to Ditto all the time, so a HTTP connection might be a better fit
    mfbatra
    @mfbatra
    @thjaeckle Thanks alot for the feedback, I did come across RQL filters but wasn't so sure if it would work,I will give it a go and try implementing it and share my progress with everyone here.
    Alexander Wellbrock
    @lionax_gitlab
    Question on metadata: when initializing a large feature with about 100 properties and using metadata extensively (about 4 properties for every leaf) the header size limits kick in and deny the request. Should I raise the limits in this case or divide the requests into multiple smaller ones? Any opinions / experience?
    Alexander Wellbrock
    @lionax_gitlab
    About the ditto (js) client: what do you think about the client integrating with vorto (through a generic model interface + implementation) in a way that you can generate and apply metadata from a vorto-model directly in the client. The vorto implementation would then take a cache implementation (memory, browser or redis?) and store models with a retention policy. The vorto / model integration also enables validation, skeleton generation transparently through the ditto client. That's my current take on the question how to tie thing and model closer together on the client side (where usually both are required for effective programming)
    Tobias Månsson
    @tobias-zeptio
    Hi. I'm having some issues with setting up a amqp 1.0 payload mapper. Is there a way to debug messages coming in, to see what's happening in the mapper?
    I get this message in connectivity "Received message from AMQP 1.0 with externalMessageHeaders: {absolute-expiry-time=1610020615816, to=iot.hdt.in}"
    So message seems to be received
    Then "Resolved mapped headers of ImmutableDittoHeaders..."
    And lastly "Acking <null> with original external message headers..."
    Florian Fendt
    @ffendt

    @lionax_gitlab :
    On metadata: As of now we don't have experience with that extensive amount of metadata. Sending an update per property seems like a lot of message (and there's probably no other way to divide the requests meaningful, otherwise it would probably be already divided into multiple features). So from my gut feeling it seems to be reasonable to increase the limit.

    on the client: you think about having this in both clients? I like the idea of having a generic model interface that could be used by users if wanted. Concrete implementations of this should probably not reside directly in the respective client, but besides as a dependency users can import to their project.

    Florian Fendt
    @ffendt
    @tobias-zeptio: So you're looking for the payload of the incoming message? I think there are two ways getting it.
    The first one would be enabling the logging feature of the connection (see Enable connection logs, read the top of the page for learning on how to send piggyback commands to Ditto).
    The second one would be setting the logging level of org.eclipse.ditto.services.connectivity.messaging.amqp.AmqpConsumerActor to DEBUG and viewing the container logs directly (see Change a specific log level for one service, see the top of the paragraph for more information on the piggyback commands).
    Alexander Wellbrock
    @lionax_gitlab
    @ffendt
    on metadata: thanks for your opinion
    on client: fair enough, I though of vorto being a good reference / default implementation since it's also packaged together in the IoT package (ditto, hono, vorto) but then again it's more general to have it as additional dependency - I like that. I'm currently only planning for the JS client, since that's the field of main interest regarding our use-cases and where the resources are put to use. The usefulness however is language independent
    Tobias Månsson
    @tobias-zeptio
    Thank you @ffendt , that helps alot!
    Tobias Månsson
    @tobias-zeptio
    So I think I've narrowed down this issue for my payloads. If I use a Java based AMQP library to send the message, it works. But I'm using rhea, which is javascript. And I think the issue is that ditto expects a JMS formated message. Is that the case?
    mfbatra
    @mfbatra

    Hi guys,
    I have been trying to create an mqtt connection to myqtthub. It provides with user credentials such as username, password and client id, But it is unable to create it:

    "uri": "tcp://<username>:<password>@node02.myqtthub.com:1883",
    "specificConfig": {
    "clientId": "<client-id>"
    }

    The other alternative i have tried is to use an sci connection but it does not seem to work with that as well where we just define the sci string in the client id as follows:

    "uri": "tcp://node02.myqtthub.com:1883",
    "specificConfig": {
    "clientId": "<sci-string>"
    }

    Can anyone please let me know what I am doing wrong?

    Thomas Jaeckle
    @thjaeckle
    @mfbatra what is the error message you see? "unable to create it" is a little unspecific
    mfbatra
    @mfbatra

    Hi @thjaeckle , Thank you so much for the response. I am able to createConnection but when i modifyConnection then i get the following response:

    { "?": { "?": { "type": "devops.responses:errorResponse", "status": 504, "serviceName": null, "instance": null, "payload": { "status": 504, "error": "connectivity:connection.failed", "message": "The Connection with ID 'mqtt-example-connection-123' failed to connect.", "description": "Ask timed out on [Actor[akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$e#-1207142866]] after [55000 ms]. Message of type [akka.routing.ConsistentHashingRouter$ConsistentHashableEnvelope]. A typical reason for AskTimeoutException is that the recipient actor didn't send a reply." } } } }

    If i retrieveConnection status, This is the response i get:

    { "?": { "?": { "type": "connectivity.responses:retrieveConnectionStatus", "status": 200, "connectionId": "mqtt-example-connection-123", "connectionStatus": "open", "liveStatus": "failed", "clientStatus": [ { "type": "client", "client": "1", "status": "failed", "statusDetails": "[CONNECTING] Connection timed out at 2021-01-14T13:32:41.280643400Z while CONNECTING. Will try to reconnect.", "inStateSince": "2021-01-14T13:32:43.283439800Z" } ], "sourceStatus": [], "targetStatus": [] } } }
    _

    Thomas Jaeckle
    @thjaeckle
    and your Ditto docker container is able to connect to the outside world?
    did you try e.g. connecting to test.mosquitto.org without authentication?
    mfbatra
    @mfbatra
    Yes, I have already tested it with mosquitto and it works.
    mfbatra
    @mfbatra
    Also i have tested the mqtt connection of mqtthub with MQTTlens and it establishes a connection with the broker so the credentials are correct as well.
    Thomas Jaeckle
    @thjaeckle
    @mfbatra did you check the logs of the Ditto connectivity service? if those doesn't help you can also increase the log level to DEBUG by configuring the environment variables LOG_LEVEL and LOG_LEVEL_APPLICATION to DEBUG
    mfbatra
    @mfbatra

    @thjaeckle When i retrieveConnectionLogs, I do not get any information from it:

    {
    "?": {
    "?": {
    "type": "connectivity.responses:retrieveConnectionLogs",
    "status": 200,
    "connectionId": "mqtt-example-connection-123",
    "connectionLogs": [],
    "enabledSince": null,
    "enabledUntil": null
    }
    }
    }

    Here are the logs of docker_connectivity:

    2021-01-15 16:59:14,050 INFO [] o.e.d.s.c.m.m.h.HiveMqtt3ClientActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$g/c1 - Publisher started. Now starting consumers.

    2021-01-15 16:59:14,053 INFO [] o.e.d.s.c.m.m.h.HiveMqtt3ClientActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$g/c1 - Consumers started. Client actor is now ready to process messages.

    2021-01-15 16:59:14,055 INFO [] o.e.d.s.c.m.m.h.HiveMqtt3ClientActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$g/c1 - Client connected and all consumers ready, subscribing now.

    2021-01-15 16:59:27,094 INFO [] o.e.d.s.u.p.m.DefaultPersistenceStreamingActor akka://ditto-cluster/user/connectivityRoot/persistenceStreamingActor - Starting stream for <SudoStreamPids [type=streaming:SudoStreamPids, dittoHeaders=ImmutableDittoHeaders [{}], burst=25, timeoutMillis=10800000, lowerBound=LowerBound [id=:_, revision=0]]>

    2021-01-15 16:59:27,123 INFO [465e929f-9441-4507-bf5e-f2170f2e0f29] o.e.d.s.c.m.p.ConnectionPersistenceActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa - Starting cleanup for 'connection:mqtt-example-connection-123', deleting snapshots to sequence number 69 and events to 69.

    2021-01-15 16:59:27,155 INFO [465e929f-9441-4507-bf5e-f2170f2e0f29] o.e.d.s.c.m.p.ConnectionPersistenceActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa - Cleanup for 'connection:mqtt-example-connection-123' completed.

    2021-01-15 17:00:08,779 WARN [465e929f-9441-4507-bf5e-f2170f2e0f29] o.e.d.s.c.m.p.ConnectionPersistenceActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa - Operation <open-connection> on connection <mqtt-example-connection-123> failed due to ConnectionFailedException: The Connection with ID 'mqtt-example-connection-123' failed to connect..

    2021-01-15 17:00:14,070 INFO [] a.r.RemoteActorRefProvider$RemoteDeadLetterActorRef akka://ditto-cluster/deadLetters - Message [akka.actor.Status$Failure] from Actor[akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$g/c1#810613972] to Actor[akka://ditto-cluster/deadLetters] was not delivered. [12] dead letters encountered, of which 1 were not logged. The counter will be reset now. If this is not an expected behavior then Actor[akka://ditto-cluster/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

    2021-01-15 17:00:14,259 INFO [] o.e.d.s.c.m.m.h.HiveMqtt3ClientActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$g/c1 - Publisher started. Now starting consumers.

    2021-01-15 17:00:14,260 INFO [] o.e.d.s.c.m.m.h.HiveMqtt3ClientActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$g/c1 - Consumers started. Client actor is now ready to process messages.

    2021-01-15 17:00:14,262 INFO [] o.e.d.s.c.m.m.h.HiveMqtt3ClientActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$g/c1 - Client connected and all consumers ready, subscribing now.

    Thomas Jaeckle
    @thjaeckle
    @mfbatra sorry, without DEBUG logs or debuggig I also have no idea what is the problem here
    Thomas Jaeckle
    @thjaeckle

    @mfbatra ok, I think I found out what is the problem here .. Ditto sets a different clientId for the MQTT publisher (using the configured clientId suffixed with a "p") - if no explicit "publisherId" was configured - whenever using a separate publisher client is configured (which is by default enabled)

    for your use-case that is not helpful beacause the "myqtthub" requires you to use a specific "clientId" - so either deactivate that separatePublisherClient configuration or configure that the "publisherId" is the same as the clientId

    mfbatra
    @mfbatra
    @thjaeckle This worked perfectly
    Bob Claerhout
    @BobClaerhout

    Hi,

    I'm trying to update ditto from version 1.4.0 to 1.5.1 using helm. However, outgoing connections don't seem to migrate properly.
    I have a working connection with a target as follows:

    {
        "?": {
            "?": {
                "type": "connectivity.responses:retrieveConnection",
                "status": 200,
                "connection": {
                    "id": "history",
                    "name": null,
                    "connectionType": "amqp-10",
                    "connectionStatus": "open",
                    "uri": "amqp://<user>:<password>@<host>:<port>",
                    "sources": [],
                    "targets": [
                        {
                            "address": "queue://history",
                            "topics": [
                                "_/_/things/twin/events?extraFields=attributes/LogicalName"
                            ],
                            "authorizationContext": [
                                "<auth>"
                            ],
                            "headerMapping": {
                                "content-type": "{{header:content-type}}",
                                "reply-to": "{{header:reply-to}}",
                                "correlation-id": "{{header:correlation-id}}"
                            }
                        }
                    ],
                    "clientCount": 1,
                    "failoverEnabled": true,
                    "validateCertificates": true,
                    "processorPoolSize": 5,
                    "tags": []
                }
            }
        }
    }

    After the update, this ditto connection doesn't work anymore. In the metrics, all counters are 0. In the connectionlog (enabled via devops), no logs appear. Both the client and the target are open (retrieved via te retrieveConnectionStatus).
    I also have another connection in the same cluster:

    {
        "?": {
            "?": {
                "type": "connectivity.responses:retrieveConnection",
                "status": 200,
                "connection": {
                    "id": "ditto",
                    "name": null,
                    "connectionType": "amqp-10",
                    "connectionStatus": "open",
                    "uri": "amqp://<user>:<password>@<host>:<port>",
                    "sources": [
                        {
                            "addresses": [
                                "ditto"
                            ],
                            "consumerCount": 1,
                            "authorizationContext": [
                                "<auth>"
                            ],
                            "headerMapping": {
                                "content-type": "{{header:content-type}}",
                                "reply-to": "{{header:reply-to}}",
                                "correlation-id": "{{header:correlation-id}}"
                            },
                            "payloadMapping": [
                                "Ditto"
                            ],
                            "replyTarget": {
                                "address": "{{header:reply-to}}",
                                "headerMapping": {
                                    "content-type": "{{header:content-type}}",
                                    "correlation-id": "{{header:correlation-id}}"
                                },
                                "expectedResponseTypes": [
                                    "error",
                                    "response"
                                ],
                                "enabled": true
                            }
                        }
                    ],
                    "targets": [],
                    "clientCount": 1,
                    "failoverEnabled": true,
                    "validateCertificates": true,
                    "processorPoolSize": 5,
                    "tags": []
                }
            }
        }
    }

    This connection just works after the update.
    Before the update, I looked at the release notes and couldn't find anything which might impact this. Any suggestions?
    I also noticed that the appVersion of the current helm release (in the packages) is set to 1.5.0. Is this on purpose? The helm chart version has been updated to 1.5.1 though.

    I rolled back the update to 1.4.0 after which the connections worked again
    Thomas Jaeckle
    @thjaeckle
    @BobClaerhout that's strange - the connection should still work
    did you see any ERROR or WARNING logs regarding that connection in the connectivity service?
    but you mention retrieveConnectionStatus to return that both connections are working .. so that's strange
    this normally means that something with the authorizationContext does not match the actually contained "read-subjects" of e.g. the events to publish
    can you enable debug logging by setting LOG_LEVEL_APPLICATION to DEBUG for the things and connectivity services?
    that way we could have a look at the emitted twin events and their "read-subjects" header in things and whether they arrive at all in connectivity
    version 1.5.1 should be used - probably we forgot to update the helm chart to use that version
    Bob Claerhout
    @BobClaerhout
    @thjaeckle thanks for the quick response. I'll try this asap and provide you the result
    Bob Claerhout
    @BobClaerhout
    surprise surprise, tried it again this morning, everything seems to be working fine now... If this pops up again, I'll ping you again.
    Bob Claerhout
    @BobClaerhout
    created a PR for updating the appVersion to 1.5.1: eclipse/packages#190
    Thomas Jaeckle
    @thjaeckle

    surprise surprise, tried it again this morning, everything seems to be working fine now... If this pops up again, I'll ping you again.

    awesome :+1:

    Aman Sharma
    @amansharma30
    I am using JAVA SDK to connect to Eclipse Ditto, all my twins and policies are being created in Sandbox Server instead of local one even though I am using http://localhost as the Host Name.
    Does anybody has any idea where this configuration is defined in the SDK?
    Thomas Jaeckle
    @thjaeckle
    @amansharma30 https://www.eclipse.org/ditto/client-sdk-java.html#instantiate--configure-a-new-ditto-client
    ...
    MessagingProvider messagingProvider =
        MessagingProviders.webSocket(WebSocketMessagingConfiguration.newBuilder()
            .endpoint("wss://ditto.eclipseprojects.io")
    ...
    Aman Sharma
    @amansharma30
    @thjaeckle Many thanks, You mean to say this would work?
    MessagingProvider messagingProvider =
    MessagingProviders.webSocket(WebSocketMessagingConfiguration.newBuilder()
    .endpoint("http://localhost")
    Aman Sharma
    @amansharma30
    OR MessagingProvider messagingProvider =
    MessagingProviders.webSocket(WebSocketMessagingConfiguration.newBuilder()
    .endpoint("wss://localhost") ?
    Thomas Jaeckle
    @thjaeckle
    When you started Ditto running on the default HTTP port 80 ws://localhost will work. When you didn't change the setup of the deployment, eg in docker Compose, Ditto however starts on port 8080, so use ws://localhost:8080instead. Note to not use wssunless you connect to a Ditto installation having a valid SSL certificate. Seems like you don't have many knowledge of such stuff, so I'm mentioning that beforehand
    Aman Sharma
    @amansharma30
    @thjaeckle Yes, Many Thanks for the valuable information.
    Tobias Månsson
    @tobias-zeptio
    Hey all. I have a question to the room. Do you use ditto as the only way for (web/cloud based) services to connect to a device, or do you have other channels as well. How do you keep them in sync in that case?
    Thomas Jaeckle
    @thjaeckle
    @tobias-zeptio we as Bosch.IO having a commercial service based on Ditto in place have that as the only way to connect to a device - other services of our Bosch IoT Suite make use of Ditto's APIs in order to communicate with devices