hello guys, I have a question .... I have finally managed to make Zipkin + Cassandra work together... however, to properly display dependencies I need to periodically run zipkin-dependencies.jar (as mentioned above) ... I do this via cron in linux ... the problem is -> it takes a little over 1hr to calc the data...
Is there any way to limit the spans to automatically check for those nonprocessed only? or any other way I can tune cassandra / zipkin-dependencies.jar to make it work faster? Any tips?
I am not sure if I should focus on the cassandra tuning or the problem is somewhere else ... I tried to run twice manually the zipkin-dependencies and it says:Running Dependencies job for 2021-03-01: 1614556800000000 =< Span.timestamp 1614643199999999 for both instances ... so it obivously recalc the same data ...
2 replies
Hi,
I'm running zipkin in a container and sending a get (wrapFetch/node-fetch) trace to it (zipkin-transport-http):
const { Tracer } = require('zipkin');
const { BatchRecorder } = require('zipkin'); // format data spans
const { HttpLogger } = require('zipkin-transport-http');
const fetch = require('node-fetch');
const wrapFetch = require('zipkin-instrumentation-fetch');
const CLSContext = require('zipkin-context-cls'); // continous local storage
const ctxImpl = new CLSContext();
const localServiceName = 'xxx';
const recorder = new BatchRecorder({
logger: new HttpLogger({
endpoint: http://localhost:9411/api/v1/spans
})
});
const tracer = new Tracer({ ctxImpl, recorder, localServiceName });
const remoteServiceName = 'youtube';
const zipkinFetch = wrapFetch(fetch, { tracer, remoteServiceName });
zipkinFetch('https://www.youtube.com/').then(res => res)
How do I go about instrumenting and sending my own span?
Thanks,
3 replies
1 reply
4 replies
Seems like none of the folding or annotation stuff is available? I'm trying opentelemetry from node, there doesn't seem to be a way to override the serviceName for the http or the db? The remoteService? Is there some way to tag spans so they are remote? Change their kind? Really don't know where to look.
3 replies
4 replies
Hi all, running into a small issue connecting Zipkin to AWS Elasticsearch. Initially I was getting EmptyEndpointGroupException then I forced the health check off and realized that Zipkin couldn't find the AWS credentials so it's throwing "User: anonymous is not authorized to perform: es:ESHttpGet". This is what I'm using to execute:
STORAGE_TYPE=elasticsearch \
ES_HOSTS=https://<domain>.us-east-1.es.amazonaws.com/ \
ES_HEALTH_CHECK_ENABLED=false \
java -Dloader.path='aws.jar,aws.jar!/lib' -Dspring.profiles.active=aws \
-cp zipkin.jar org.springframework.boot.loader.PropertiesLauncher
I've even tried to forcing the environment variables but still same issue:
STORAGE_TYPE=elasticsearch \
ES_HOSTS=https://<domain>.us-east-1.es.amazonaws.com/ \
ES_HEALTH_CHECK_ENABLED=false \
AWS_ACCESS_KEY=<access> \
AWS_SECRET_KEY=<secret> \
java -Dloader.path='aws.jar,aws.jar!/lib' -Dspring.profiles.active=aws \
-cp zipkin.jar org.springframework.boot.loader.PropertiesLauncher
5 replies
1 reply
Hello folks, we are seeing some fatal error: concurrent map read and map write errors in our main application process and further debugging shows some issue with Zipkin library: github.com/openzipkin/zipkin-go
goroutine 182 [running]:
runtime.throw(0xe75af5, 0x21)
/usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc1c015f420 sp=0xc1c015f3f0 pc=0x4313f2
runtime.mapaccess2(0xcf7160, 0xc262f27e30, 0xc19f6391f0, 0xc19f6391f0, 0xc17eac231b)
/usr/local/go/src/runtime/map.go:470 +0x278 fp=0xc1c015f468 sp=0xc1c015f420 pc=0x40fc58
reflect.mapaccess(0xcf7160, 0xc262f27e30, 0xc19f6391f0, 0xe6a8ed)
/usr/local/go/src/runtime/map.go:1319 +0x3f fp=0xc1c015f4a0 sp=0xc1c015f468 pc=0x411f6f
reflect.Value.MapIndex(0xcf7160, 0xc1300544d0, 0x195, 0xcbb920, 0xc19f6391f0, 0x98, 0xcbb920, 0xc19f639250, 0x98)
/usr/local/go/src/reflect/value.go:1176 +0x16d fp=0xc1c015f518 sp=0xc1c015f4a0 pc=0x49771d
. . .
github.com/openzipkin/zipkin-go/reporter.JSONSerializer.Serialize(...)
/src/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.2.2/reporter/serializer.go:36
github.com/openzipkin/zipkin-go/reporter.(*JSONSerializer).Serialize(0x164b018, 0xc1951c4480, 0x64, 0x90, 0x0, 0xc0e90ac380, 0x433160, 0xea6f90, 0xc000056a00)
<autogenerated>:1 +0x72 fp=0xc1c015fe78 sp=0xc1c015fe30 pc=0xb6f892
github.com/openzipkin/zipkin-go/reporter/http.(*httpReporter).sendBatch(0xc0000293b0, 0xc1c015ffaf, 0x1)
/src/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.2.2/reporter/http/http.go:137 +0xb2 fp=0xc1c015ff90 sp=0xc1c015fe78 pc=0xb8a882
github.com/openzipkin/zipkin-go/reporter/http.(*httpReporter).sendLoop(0xc0000293b0)
/src/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.2.2/reporter/http/http.go:99 +0x3f fp=0xc1c015ffd8 sp=0xc1c015ff90 pc=0xb8a49f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc1c015ffe0 sp=0xc1c015ffd8 pc=0x4615c1
created by github.com/openzipkin/zipkin-go/reporter/http.NewReporter
/src/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.2.2/reporter/http/http.go:254 +0x2dbThis is interesting as it is an error while serializing a Span for the Zipkin client. Upon closer inspection, there is a Tag map in the spans that seems a candidate for the concurrency issue. If the Span is being serialized and a new Tag or something is being added to it concurrently, we may run into an issue.
Does any of you have seen this issue or are aware of it? Any help would be appreciated.
Regards.
3 replies
Hello!
I wanted to open a possible bug issue but came to ask here first.
I am using Spring Boot, Sleuth and Brave. There is a weird race condition that pops up and we are stumped.
The issue investigation is described here spring-cloud/spring-cloud-sleuth#1814 where Brave throws an exception "Parent can only be null in a local root". In the last message (https://github.com/spring-cloud/spring-cloud-sleuth/issues/1814#issuecomment-767389228) is the "stumped" part. In debug another subsequent call to the same method works fine.
Any help would be much appreciated.
6 replies
Is it possible to use custom id as a trace id in zipkin ?
hello guys, I just have a question ... I have zipkin running together with cassandra as its persistent storage .. the issue I have is that with the external storage I have to recalc dependencies periodically .. it takes a bit over an hour to get that done ....
Is there the same issue with ElasticSearch ??? or is it faster? or do you have any tips how to speed it up??
I’m not sure this is the right place to talk about this. So, Looking forward to your reply. Thanks a lot!
1 reply
[{"traceId":"607e3e086301ffaf74e217d5983efbc5","parentId":"74e217d5983efbc5","id":"91d3f96d11d7cb8e","kind":"SERVER","name":"get /api/msgcenter/msg/lastest","timestamp":1618886163861028,"duration":3775,"localEndpoint":{"serviceName":"base-msg-center","ipv4":"10.20.5.160"},"remoteEndpoint":{"ipv4":"123.157.238.102"},"tags":{"http.method":"GET","http.path":"/api/msgcenter/msg/lastest","mvc.controller.class":"ClientAPIController","mvc.controller.method":"getLastestMsgRecord"},"shared":true},{"traceId":"607e3e086301ffaf74e217d5983efbc5","parentId":"74e217d5983efbc5","id":"91d3f96d11d7cb8e","kind":"CLIENT","name":"get","timestamp":1618886152337857,"duration":11526274,"localEndpoint":{"serviceName":"lingyue-gateway","ipv4":"10.20.6.52"},"tags":{"http.method":"GET","http.path":"/api/msgcenter/msg/lastest"}},{"traceId":"607e3e086301ffaf74e217d5983efbc5","parentId":"74e217d5983efbc5","id":"864bdbd5a06a0515","kind":"SERVER","name":"get /auth/check","timestamp":1618886152333018,"duration":5224,"localEndpoint":{"serviceName":"dmo-user","ipv4":"10.20.7.38"},"remoteEndpoint":{"ipv4":"10.20.6.52","port":33926},"tags":{"http.method":"GET","http.path":"/auth/check","mvc.controller.class":"AuthController","mvc.controller.method":"checkToken"},"shared":true},{"traceId":"607e3e086301ffaf74e217d5983efbc5","id":"74e217d5983efbc5","localEndpoint":{"serviceName":"lingyue-gateway","ipv4":"10.20.6.52"},"annotations":[{"timestamp":1618886174659000,"value":"brave.flush"}]},{"traceId":"607e3e086301ffaf74e217d5983efbc5","parentId":"74e217d5983efbc5","id":"864bdbd5a06a0515","kind":"CLIENT","name":"get","timestamp":1618886152331773,"duration":5668,"localEndpoint":{"serviceName":"lingyue-gateway","ipv4":"10.20.6.52"},"tags":{"http.method":"GET","http.path":"/auth/check"}},{"traceId":"607e3e086301ffaf74e217d5983efbc5","id":"74e217d5983efbc5","kind":"SERVER","name":"get","timestamp":1618886152322007,"duration":11542205,"localEndpoint":{"serviceName":"lingyue-gateway","ipv4":"10.20.6.52"},"remoteEndpoint":{"ipv4":"123.157.238.102"},"tags":{"http.method":"GET","http.path":"/base-msg-center/api/msgcenter/msg/lastest"}}]
2 replies
Hi all,I have deployed Jaeger on istio to monitor the bookinfo application. Then I want to analyze the trace data of Jaeger. However, I am confused by some span tags, such as, upstream_cluster and peer address. I found the meaning of peer.address from https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk_exporters/zipkin.md
, It says that peer.address means remote address. I think it means that when a client start a request and a server receive the request, the client's peer.address will be the server and the server's peer address will be the client.
Here is some information I get from one trace with 8 spans.
spanID operationName podIP serviceName pid refType peer.address span.kind
0 0044bea01e572833 productpage.default.svc.cluster.local:9080/pro... 10.244.27.204 istio-ingressgateway 192.168.0.49 client
1 a2bdd3593ab25d00 productpage.default.svc.cluster.local:9080/pro... 10.244.27.238 productpage.default 0044bea01e572833 CHILD_OF 10.244.27.204 server
2 626e09f9758da789 details.default.svc.cluster.local:9080/* 10.244.27.238 productpage.default a2bdd3593ab25d00 CHILD_OF 10.244.27.238 client
3 5a76f143f0e5c301 details.default.svc.cluster.local:9080/* 10.244.27.247 details.default 626e09f9758da789 CHILD_OF 10.244.27.238 server
4 9c1aaf91ea5e5ce9 reviews.default.svc.cluster.local:9080/* 10.244.27.238 productpage.default a2bdd3593ab25d00 CHILD_OF 10.244.27.238 client
5 923db83990a9e7e5 reviews.default.svc.cluster.local:9080/* 10.244.27.246 reviews.default 9c1aaf91ea5e5ce9 CHILD_OF 10.244.27.238 server
6 8e3dbd4dd0b70f85 ratings.default.svc.cluster.local:9080/* 10.244.27.246 reviews.default 923db83990a9e7e5 CHILD_OF 10.244.27.246 client
7 f227decb8d8b2c51 ratings.default.svc.cluster.local:9080/* 10.244.27.217 ratings.default 8e3dbd4dd0b70f85 CHILD_OF 10.244.27.246 serverHowever, the peer.address seems to be it's parent span's pod id, which is different from what I understand, and I don't know which is right.
Besides, I don't understand the meaning of upstream_cluster and downstream_cluster as below. Why all downstream_cluster is None, and what is the meaning of '9080' and 'inbound' in outstream_cluster?
spanID operationName podIP ... upstream_cluster downstream_cluster span.kind
0 0044bea01e572833 productpage.default.svc.cluster.local:9080/pro... 10.244.27.204 ... outbound|9080||productpage.default.svc.cluster... - client
1 a2bdd3593ab25d00 productpage.default.svc.cluster.local:9080/pro... 10.244.27.238 ... inbound|9080|| - server
2 626e09f9758da789 details.default.svc.cluster.local:9080/* 10.244.27.238 ... outbound|9080||details.default.svc.cluster.local - client
3 5a76f143f0e5c301 details.default.svc.cluster.local:9080/* 10.244.27.247 ... inbound|9080|| - server
4 9c1aaf91ea5e5ce9 reviews.default.svc.cluster.local:9080/* 10.244.27.238 ... outbound|9080||reviews.default.svc.cluster.local - client
5 923db83990a9e7e5 reviews.default.svc.cluster.local:9080/* 10.244.27.246 ... inbound|9080|| - server
6 8e3dbd4dd0b70f85 ratings.default.svc.cluster.local:9080/* 10.244.27.246 ... outbound|9080||ratings.default.svc.cluster.local - client
7 f227decb8d8b2c51 ratings.default.svc.cluster.local:9080/* 10.244.27.217 ... inbound|9080|| - server
1.Unsatisfied dependencies for type HttpTracing with qualifiers @Default
at injection point [BackedAnnotatedParameter] Parameter 1 of [BackedAnnotatedConstructor] @Inject brave.jaxrs2.TracingClientFilter(HttpTracing)
2.Unsatisfied dependencies for type ContainerParser with qualifiers @Default
at injection point [BackedAnnotatedParameter] Parameter 1 of [BackedAnnotatedConstructor] @Inject brave.jaxrs2.SpanCustomizingContainerFilter(ContainerParser)
How i must define HttpTracing and ContainerParser beans?
Thank you.
Hi!
At our company we're going to implement Zipkin. However as database we're obliged to use PostgreSQL. Is there any documentation regarding the database component so that we could write our own implementation?
I checked out the Zipkin source code/javadoc and also looked at the Cassandra and ElasticSearch implementations but some kind of documentation would be very helpful.
Maybe other remarks that we should take into account? Or tips to make this implementation easier?
Thanks in advance for the info!
Kr
2 replies
- I am running dockerized zipkin with mysql as storage. It also runs zipkin-dependencies looking at docker-compose file. To run MySQL, was it necessary to run zipkin-dependencies?
- How can I manually call zipkin-dependencies for aggregration purpose while running as a dependency in 1. Or does it runs automatically when a data is sent to zipkin? Or does run as stated end of the day?
1 reply
Golang seemed more protobuf encoder friendly, so I created a small example application along the lines of "brave-example", "zipkin-php-example", & "zipkin-js-example". If anyone is interested or has feedback, the repo is here: https://github.com/smlobo/zipkin-go-example
4 replies
3 replies
I was typing up a question here about how I could include Zipkin as part of my Spring Boot project as a subproject (Gradle) without using EnableZipkinServer, Docker Compose or the self-contained jar since the compose setup has problems with live-reload and cache sharing, while the self contained jar would've been available with Gradle ideally to avoid unnecessary binaries in the repo. I soon answered my own question after thinking it through, so I wanna document it for archival's sake even if it may be almost obvious to someone more experienced:
Treat Zipkin as an external service, and use Docker Compose to describe your external services. This also includes things like databases, messaging servers and cache servers. Use Docker Compose to ready the infrastructure and let your build tool manage your code outside Docker instead. No custom binaries for your external dependencies, no manual cache management, and as an added bonus, single command infrastructure.
I sure would've loved to grok this earlier so I didn't have to spend so many days trying different options. Ah well, sometimes you learn the hard way xD
1 reply