https://www.eclipse.org/ditto/2021-05-06-release-announcement-200.html
Thank you for all your support and especially to your (code) contributions which made it into this huge release
if you have access to the MongoDB backing Ditto, you can find the historical state values in the MongoDB collection
things_journalby default, this collection is however cleaned up periodically - if you don't want that clean-up you may configure the environment variable
PERSISTENCE_CLEANUP_ENABLED of the "concierge" service to false, see also:https://github.com/eclipse/ditto/blob/2.0.1/concierge/service/src/main/resources/concierge.conf#L71
Greetings community, let's see if you can help me with a problem I have.
I have followed this guide (https://github.com/eclipse/ditto-examples/tree/master/mqtt-bidirectional) to connect Ditto, Mosquitto and a device using the MQTT protocol. It has been a total success and I am already receiving messages from the defined topic.
The problem I have is with a feature of my device. This device transmits a very heavy type of data (you can think of a base64 encoded audio, image or video) and it shows this exception in the ditto-connectivity microservice:
2021-05-20 16:29:46,171 INFO [31343174-6640-4ebd-a842-4009660f2ef1] o.e.d.c.s.m.InboundDispatchingActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$a/c1/inboundDispatching - Got exception <things:thing.toolarge> when processing external message with mapper <javascript>: <The size of '257.071' kB exceeds the maximal allowed Thing size of '100' kB.>Of course, Ditto has a kiB limit on its messages that can be configurable as written here: https://www.eclipse.org/ditto/release_notes_080-M2.html#changes
The problem is that I have already configured the ditto-limits.conf file this way ...
# limitations regarding "Thing" entity / "things" service
things {
max-size = 500k
}
...
# limitations regarding "messages" (e.g. via WebSocket, connectivity)
messages {
max-size = 500k
headers-size = 5k
headers-size = ${?LIMITS_MESSAGES_HEADERS_SIZE}
auth-subjects-count = 100
auth-subjects-count = ${?LIMITS_MESSAGES_AUTH_SUBJECTS_COUNT}
}and the ditto-akka-config.conf this way:
advanced {
# Maximum serialized message size, including header data. # default: 256 KiB
maximum-frame-size = 1256 KiB
maximum-frame-size = ${?REMOTE_MAX_FRAMESIZE}
# Direct byte buffers are reused in a pool with this maximum size.I have deleted the containers, I have deployed it again with docker with this files edited and it still does not work... I do not know if I am doing something wrong... could it be that I am changing the value in the file instead of using system variables with export? I have tried to do export LIMITS_THINGS_MAX_SIZE=500 and after that do the docker compose but it hasn't worked...
And, since I'm asking this, how do you see the approach of using base64 to encode multimedia? Is there a better approach to encode multimedia content for a Ditto Thing? Thank you very much for reading me ^. ^
@amable_alisys_gitlab how did you done last step in this tutorial where you define message converter for mqtt on ditto. I only get error in parsing function.
I have this i guess:
curl -X POST http://localhost:8080/devops/piggyback/connectivity?timeout=10 -u 'devops:foobar' -H 'Content-Type: application/json' -d '{
"targetActorSelection": "/system/sharding/connection",
"headers": {
"aggregate": false
},
"piggybackCommand": {
"type": "connectivity.commands:modifyConnection",
"connection": {
"id": "mqtt-example-connection-123",
"connectionType": "mqtt",
"connectionStatus": "open",
"failoverEnabled": true,
"uri": "tcp://test.mosquitto.org:1883",
"sources": [{
"addresses": ["ditto-tutorial/#"],
"authorizationContext": ["nginx:ditto"],
"qos": 0,
"filters": []
}],
"targets": [{
"address": "ditto-tutorial/{{ thing:id }}",
"topics": [
"_/_/things/twin/events",
"_/_/things/live/messages"
],
"authorizationContext": ["nginx:ditto"],
"qos": 0
}],
"mappingContext": {
"mappingEngine": "JavaScript",
"options": {
"incomingScript": "function mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {const jsonString = String.fromCharCode.apply(null, new Uint8Array(bytePayload));const jsonData = JSON.parse(jsonString); const thingId = jsonData.thingId.split(':'); const value = { temp_sensor: { properties: { value: jsonData.temp } },altitude: { properties: { value: jsonData.alt }}}; return Ditto.buildDittoProtocolMsg(thingId[0], thingId[1], 'things', 'twin', 'commands', 'modify', '/features', headers, value);}"
}
}
}
}
}'And respose:
{"?":{"?":{"type":"devops.responses:errorResponse","status":400,"serviceName":null,"instance":null,"payload":{"status":400,"error":"connectivity:connection.configuration.invalid","message":"The message mapper configuration failed due to: syntax error (incomingScript#1) - in line/column #1/242, source:\nfunction mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {const jsonString = String.fromCharCode.apply(null, new Uint8Array(bytePayload));const jsonData = JSON.parse(jsonString); const thingId = jsonData.thingId.split(:); const value = { temp_sensor: { properties: { value: jsonData.temp } },altitude: { properties: { value: jsonData.alt }}}; return Ditto.buildDittoProtocolMsg(thingId[0], thingId[1], things, twin, commands, modify, /features, headers, value);}","description":"Check the configuration options of your mapper for errors."}}}}Oh, yes, let me explain:
You must send that petition as you do, that is fine. But the code on the "incomingScript" must be a javascript code without errors/bugs.
As is all in one line is easy to have a mistake, i recomend you to use an GUI like Visual Studio or other to program the javascript code and then write it on the petition.
In your case i see you have put
const thingId = jsonData.thingId.split(:);and it must be
const thingId = jsonData.thingId.split(':');the same with:
return Ditto.buildDittoProtocolMsg(thingId[0], thingId[1], things, twin, commands, modify, /features, headers, value);}It must be something like:
return Ditto.buildDittoProtocolMsg(thingId[0], thingId[1], 'things', 'twin', 'commands', 'modify', '/features', headers, value);to test it I have used Visual Studio Code ^.^
@amable_alisys_gitlab Oh tnx i see it. Now it works yay. Can you say how you send message to mqtt broker ?
I try to update value but noting happen.
Guess this is right payload ?
mosquitto_pub -h test.mosquitto.org -t /ditto-tutorial/my.test:octopus -m '{ "temp": 30, "alt": 360, "thingId": "my.test:octopus" }'
@Canadianboy122_twitter your request seen fine, but i can't be sure because to make it run I have implemented a little client on NodeJS that emulates a real device.
(I have had to remove some parts of the code, sorry. This code is more or less functional and would have to be tweaked to make it work, but it is a good schema that lets you see what I'm doing)
var mqtt = require('mqtt');
var client = mqtt.connect("mqtt://test.mosquitto.org:1883");
console.log("connected flag " + client.connected);
client.on("error", function(error) { console.log("Can't connect. Error: " + error); process.exit(1); });
client.on("connect", function(){
console.log("connected " + client.connected);
});
//handle incoming messages
client.on('message',function(topic, message, packet) {
console.log("Message received:");
console.log("\tmessage is: ");
console.log(message);
console.log("\ttopic is: "+ topic);
});
//publish
function publish(topic,msg,options){
console.log("publishing",msg);
if (client.connected == true) {
client.publish(topic,msg,options);
}
count+=1;
}
var topic = "ditto-tutorial/my.test:octopus"
var json = { temp: 30.77, alt: 359.341, thingId: "my.test:octopus" }
var count = 0;
// I subscribe myself to check on console the message send
client.subscribe(topic, { qos:1 }); // single topic
var timer_id = setInterval(
function(){
json.alt= json.alt + 1;
publish(topic, JSON.stringify(json), options);
}, 5000);If you want to follow this aproach and implement a mqtt client on NodeJS, i have follow this guide and perhaps it can help you:
http://www.steves-internet-guide.com/using-node-mqtt-client/
- I am on Ubuntu 18.04
- I take the project with
git clone --depth 1 https://github.com/eclipse/ditto - I do cd on /ditto/deployment/docker directory.
- I use export <VAR = Value> with all and if i do
export -pI can see the variables declared:declare -x DITTO_EXTERNAL_PORT="80" declare -x HOME="/home/elgestor" declare -x LANG="C.UTF-8" declare -x LESSCLOSE="/usr/bin/lesspipe %s %s" declare -x LESSOPEN="| /usr/bin/lesspipe %s" declare -x LIMITS_MESSAGES_MAX_SIZE="1000k" declare -x LIMITS_POLICIES_MAX_SIZE="500k" declare -x LIMITS_THINGS_MAX_SIZE="500k" declare -x LOGNAME="elgestor" declare -x PWD="/home/elgestor/ditto/deployment/docker" declare -x REMOTE_MAX_FRAMESIZE="1256 KiB" declare -x SHELL="/bin/bash" declare -x SHLVL="1" declare -x SSH_TTY="/dev/pts/0" declare -x TERM="xterm" declare -x USER="elgestor" declare -x XDG_DATA_DIRS="/usr/local/share:/usr/share:/var/lib/snapd/desktop" declare -x XDG_RUNTIME_DIR="/run/user/1000" declare -x XDG_SESSION_ID="652" - I use
docker-compose up -dto build the containers. - All deployed and fine.
- I do the
sudo docker attach docker_connectivity_1to see what is loging the connectivity container. All seens fine. - I send the MQTT message with 257 Kb (an base64 image) and the connection docker shows this log:
2021-05-24 09:54:45,418 INFO [] o.e.d.c.s.m.InboundDispatchingActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$a/c1/inboundDispatching - Resolved mapped headers of ImmutableDittoHeaders [{mqtt.qos=0, mqtt.retain=true, ditto-entity-id=thing:camera:camera, mqtt.topic=ditto-tutorial/camera:camera, response-required=false, ditto-reply-target=0, ditto-expected-response-types=["error","response"], ditto-origin=mqtt-example-connection-123, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["nginx:ditto"]}, correlation-id=9a9b1454-943d-4f2c-81bb-71d5f17a39fe}] : with HeaderMapping Optional[ImmutableHeaderMapping [mapping={}]] : and external headers {mqtt.qos=0, mqtt.retain=true, mqtt.topic=ditto-tutorial/camera:camera} 2021-05-24 09:54:47,350 INFO [5ae7a31d-df8f-4022-ad15-be2ad2ce56bd] o.e.d.c.s.m.InboundDispatchingActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$a/c1/inboundDispatching - Got exception <things:thing.toolarge> when processing external message with mapper <javascript>: <The size of '257.071' kB exceeds the maximal allowed Thing size of '100' kB.> 2021-05-24 09:54:47,350 INFO [] o.e.d.c.s.m.InboundDispatchingActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$a/c1/inboundDispatching - Resolved mapped headers of ImmutableDittoHeaders [{mqtt.qos=0, mqtt.retain=true, ditto-entity-id=thing:camera:camera, mqtt.topic=ditto-tutorial/camera:camera, response-required=false, ditto-reply-target=0, ditto-expected-response-types=["error","response"], ditto-origin=mqtt-example-connection-123, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["nginx:ditto"]}, correlation-id=5ae7a31d-df8f-4022-ad15-be2ad2ce56bd}] : with HeaderMapping Optional[ImmutableHeaderMapping [mapping={}]] : and external headers {mqtt.qos=0, mqtt.retain=true, mqtt.topic=ditto-tutorial/camera:camera} 2021-05-24 09:54:47,398 INFO [] o.e.d.i.u.p.m.DefaultPersistenceStreamingActor akka://ditto-cluster/user/connectivityRoot/persistenceStreamingActor - Starting stream for <SudoStreamPids [type=streaming:SudoStreamPids, dittoHeaders=ImmutableDittoHeaders [{}], burst=25, timeoutMillis=10800000, lowerBound=LowerBound [id=:_, revision=0]]> elgestor@vmRobotica01:~/ditto/deployment/docker$ - The error is repeated until it breaks the container as seen in log.
@amable_alisys_gitlab Why you think its error ? It looks like INFO logs, not ERROR, what is problem with that logs ?
Oh i see now size problem hmmm if i find something i will edit this message.
And i have question about messages if anyone can help. Is there any example how should Thing answer on some question. There are some examples with coffeeBrewer but did not see code how Things answer on them.
https://www.eclipse.org/ditto/protocol-specification-things-messages.html
Hi @thjaeckle, thanks for your response. It is resolved; i have put the enviromental variable on the docker-compose.yml instead of put it on the system and all is ok. I had put the variables in the system because I was also defining the port on which ditto was listening as a variable in the system itself, but as you say it makes sense to put it in the docker compose.
However, I feel obliged to contact you again... Once solved, when trying to transmit an image (a 1280x720 image of 138 Kb - https://i.pinimg.com/originals/7c/2f/4b/7c2f4bfbaa411a9ef5b45bd0b4214fba.jpg) using MQTT I have received this error message on the log of docker_connectivity_1:
2021-05-25 08:58:06,047 INFO [1eb155dc-7f6b-4e81-8cb9-867f338f37e5] o.e.d.i.m.a.AcknowledgementAggregatorActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$a/c1/inboundDispatching/ackr13-1eb155dc-7f6b-4e81-8cb9-867f338f37e5 - Stopped waiting for acknowledgements because of ditto runtime exception <org.eclipse.ditto.base.model.signals.commands.exceptions.GatewayInternalErrorException [message='There was a rare case of an unexpected internal error.', errorCode=gateway:internalerror, httpStatus=HttpStatus [code=500, category=SERVER_ERROR], description='Please contact the service team.', href=null, dittoHeaders=ImmutableDittoHeaders [{}]]>.
2021-05-25 08:58:06,050 INFO [] o.e.d.c.s.m.OutboundMappingProcessorActor akka://ditto-cluster/system/sharding/connection/1/mqtt-example-connection-123/pa/$a/c1/outboundMappingProcessor - Got DittoRuntimeException 'gateway:internalerror' when ExternalMessage was processed: There was a rare case of an unexpected internal error. - Please contact the service team.As it says to contact to the service team, i contact you jejeje.
I have transmited a low resolution image too (a 326x437 image of 10 Kb - https://images-na.ssl-images-amazon.com/images/I/41n1HcdPJ%2BL._AC_SX355_.jpg) on base64 and all is fine; it appears as saved on the Ditto Thing and i can extract it without problem. I have used the same code to transmit both images, so the problem is the data transmited.
I have modify this as docker-compose.yml:
connectivity:
image: docker.io/eclipse/ditto-connectivity:${DITTO_VERSION:-latest}
mem_limit: 384m
networks:
default:
aliases:
- ditto-cluster
depends_on:
- policies
- concierge
environment:
- TZ=Europe/Berlin
- INSTANCE_INDEX=1
- BIND_HOSTNAME=0.0.0.0
- OPENJ9_JAVA_OPTIONS=-XX:+ExitOnOutOfMemoryError -Xtune:virtualized -Xss512k -XX:MaxRAMPercentage=80 -Dakka.coordinated-shutdown.exit-jvm=on -Dakka.cluster.shutdown-after-unsuccessful-join-seed-nodes=120s
- MONGO_DB_HOSTNAME=mongodb
# Set additional configuration options here
# -Dditto.connectivity...
- LIMITS_THINGS_MAX_SIZE=500k
- LIMITS_POLICIES_MAX_SIZE=500k
- LIMITS_MESSAGES_MAX_SIZE=1000k
- REMOTE_MAX_FRAMESIZE=1256KiB
command: java -jar starter.jarIf you need more data about the error I will be happy to provide it ^.^
I tried to upgrade from 1.5 to 2.0.1 but I'm experiencing a severe performance drop in the gateway service. It's load rises by approx. 300% and it starts to print WARN messages about slow HTTP connections. Any idea how to troubleshoot this or even what could be the cause?
I also notice new JWT parser errors (eclipse/ditto#1073) but I'm not sure if that is related. Sometimes requests come through and are processed as expected but most of the time from the client side of things I only see 503 errors. If this is a bug I'll open an issue but maybe it's my specific setup / config and I just don't see cause
could you maybe activate debug logging and with that hopefully find out which code prints a lot of the same messages?
I also don't think that #1073 is related, but thx for reporting
the same can already be done with policies: https://www.eclipse.org/ditto/protocol-examples-creatething.html#creatething-with-copied-policy-by-policy-id
"_copyPolicyFrom" is missing .. only some examples show that
maybe another idea: could you put in the "definition" the thingId of a thing containing the model information and reference that from all things sharing the "same" model?
Hi!
I must ask a "Thing" (hahaha): Is there a limit on the MQTT and HTTP request size? Could i, for example, put on the environment variables the LIMITS_THINGS_MAX_SIZE, LIMITS_POLICIES_MAX_SIZE, LIMITS_MESSAGES_MAX_SIZE, REMOTE_MAX_FRAMESIZE as GB of data? or there is a limit?
I am trying to send a thing of 2 MB of data by HTTP and i had the "413 Payload Too Large" response. I have put on the nginx configuration file a line client_max_body_size 0;, but this error keep appearing.
@amable_alisys_gitlab hey there! While you sure can increase those limits you'll hit hard limits eventually since those limits ensure that data sent through the ditto cluster does not become too huge for the cluster to manage. You could end up having huge performance issues increasing those limits to much. If you have to get larger amounts of data through the cluster I found that finding ways to get the data "around" the cluster are much more viable. For an example to by-pass the cluster you can have a look at https://www.eclipse.org/ditto/advanced-data-by-pass.html
A more straight forward approach would be trying to find an architecture for your use-case in which clients don't have to request data through the ditto cluster
Using a template / model / prototype thing with a link in the definition field could be a way, I have to think about that for a bit
the HTTP "413" most likely comes from Akka HTTP config which Ditto configures to 1 MB:
https://github.com/eclipse/ditto/blob/master/gateway/service/src/main/resources/gateway.conf#L393
I however wonder if in Ditto's Thing a "normalization" to a unit defined in the definition should be applied then .. the structure is also "normalized" to attributes and features (e.g. by payload mappers)
However, we are going to try to transmit a device with a lot of sensorics and with cameras. It will be around 1-2 Mb of data.
I don't think I will have performance issues with a 2 Mb max size, but I will keep an eye on the metrics of our machines. I will also study more carefully the Data By-Pass Pattern. We are doing something in that style and i hadn't seen the page you paste before. Thanks.
For the cameras, each point in time we take the image that is displayed at that moment and save it in ditto as base64. I wanted to ask you how you see this approach and if this could give a performance problem or if there is a better method to deal with the data from cameras or microphones (multimedia content in general) with Ditto
Thank you very much to both of you!