openzipkin/zipkin

Zipkin is a distributed tracing system

Koushik Arvapally

@KArvapelli_twitter

@anuraaga HI I am trying to do authorization for WEB UI , for API's I'm able do the authorization api-key, samething I'm trying to do with UI , can you please help me with UI changes ? or any other way to Protect the API's ,

Tried with adding zipkin security jar but getting

Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanDefinitionStoreException: Failed to parse configuration class [zipkin.module.security.ZipkinSecurityModule]; nested exception is java.lang.IllegalStateException: Failed to introspect annotated methods on class zipkin.module.security.ZipkinSecurityModule

Maziz

@MazizEsa

Hi all

I've encounter 404 error when sending span to new relic "https://trace-api.newrelic.com/trace/v1/api/v2/spans", 6 hours back. It was working before. Any ideas?

27 replies

The /api/v2/spans suffixes by zipkin RestTemplate . So that's why Im asking here.

arunlepuru

@arunlepuru

Hi all,
Can any one guide me please how to trace all the data like database called from controller to JPA repository in zipkin spring boot client application

Roussi Abdelghani

@roussi

Hi all,
I got Unable to establish connection to RabbitMQ server: Connection refused when trying to run zipkin-server with rabbitmq and elasticsearch, using this docker-compose file :

  version: '3.7'
  services:
    elasticsearch:
      image: elasticsearch:7.10.1
      container_name: elasticsearch
      restart: on-failure
      ports:
        - 9200:9200   
      environment: 
        #not recommended in production
         - "ES_JAVA_OPTS=-Xms750m -Xmx750m"   
         - discovery.type=single-node
    rabbitmq:
      image: rabbitmq:3.8-management-alpine
      container_name: rabbitmq
      restart: on-failure
      ports:
      - 5672:5672
      - 15672:15672
      environment:
        RABBITMQ_DEFAULT_VHOST: ${rabbitmq_vhost}
        RABBITMQ_DEFAULT_USER: ${rabbitmq_user}
        RABBITMQ_DEFAULT_PASS: ${rabbitmq_pass}
        RABBITMQ_VM_MEMORY_HIGH_WATERMARK: 1024MiB
    zipkin-server:
      image: openzipkin/zipkin:2
      container_name: zipkin-server
      depends_on: 
      - rabbitmq
      - elasticsearch
      restart: on-failure
      ports:
      - 9411:9411
      environment:
        RABBIT_CONCURRENCY: 1
        RABBIT_CONNECTION_TIMEOUT: 60000
        RABBIT_QUEUE: zipkin
        RABBIT_ADDRESSES: ${rabbitmq_host}
        RABBIT_PASSWORD: ${rabbitmq_pass}
        RABBIT_USER: ${rabbitmq_user}
        RABBIT_VIRTUAL_HOST: ${rabbitmq_vhost}
        RABBIT_USE_SSL: "false"
        STORAGE_TYPE: elasticsearch
        ES_HOSTS: ${elasticsearch_host}
      healthcheck:
        test: wget --no-verbose --tries=1 --spider http://localhost:9411/health  || exit 1
        interval: 10s
        start_period: 15s
        retries: 3
    zipkin-dependencies:
      image: openzipkin/zipkin-dependencies
      container_name: zipkin-dependencies
      depends_on: 
      - elasticsearch
      restart: on-failure
      environment:
        STORAGE_TYPE: elasticsearch
        ES_HOSTS: ${elasticsearch_host}
  volumes:
    default:
      external:
        name: zipkin-monitoring
  networks:
    default:
      external:
        name: zipkin-monitoring

knowing that rabbitmq and elastic starts normaly

2 replies

Stephen Hong

@stephenhong

Hello team, I need some clarification on remoteEndpoint. Zipkin's documentation says it's an RPC (or messaging) span, indicating the other side of the connection, but I'm not sure of what this means. There is an ipv4 field under remoteEndpoint. Can it be an IP address of an external user?

2 replies

bhogiudaykiran

@bhogiudaykiran

Hi Team, I am new here and to Zipkin as well. I am trying to start zipkin server by running Jar. can some one share the available arguments that i can pass to java -jar command? Also is there a shell script or anything handy that i can use. thanks!

4 replies

Archimedes Trajano

@trajano

Hi how do you lmit the size of Cassandra database used by Zipkin? https://stackoverflow.com/questions/66327574/how-do-you-limit-the-number-of-spans-stored-in-zipkin-server-when-using-cassandr

Dawid Węgliński

@infarot

Hi team, is there any docs about adding custom headers as zipkin tags?

8 replies

SoftHaus IT

@ITSoftHaus

Hello, somebody help me to resolve this problem:

2021-02-25 12:51:30.776 WARN [/] 1 --- [orker-epoll-2-4] c.l.a.c.e.h.HttpHealthChecker : [creqId=485fac28, chanId=b5fb1c00, laddr=172.21.0.14:36804, raddr=elasticsearch/172.21.0.4:9200][http://elasticsearch:9200/_cluster/health#GET] Unexpected 4xx health check response: [:status=401, www-authenticate=Basic realm="security" charset="UTF-8", content-type=application/json; charset=UTF-8, content-length=411] A 4xx response generally indicates a misconfiguration of the client. Did you happen to forget to configure the HealthCheckedEndpointGroup's client options?

TKS

Arun Gopalpuri

@arun0009

How can I pass custom OKHttpClient (I want to be able to configure proxy, logging interceptor etc) to OkHttpSender? I submitted a PR as I couldn’t find a way to configure proxy on OKHttpSender openzipkin/zipkin-reporter-java#200

Petr Hybler

@petr-hybler

hello guys, another thing ... I was able to manage to run the zipkin and was able to store traces successfully in Cassandra.. but now it doesn't show any dependencies .. traces are all good, accurate and "realtime" but when I click on dependencies it's empty.. .I read I have to set the zipkin-dependencies.jar to be run periodically so I did (based on this article https://stackoverflow.com/questions/37459261/cant-view-any-dependencies-inside-zipkin-ui-dependencies-tab ) but hell - it doesn't show anything still.. I even tried to run that as a service .. but no luck either ...

my zipkin service:
[Unit]
Description=Managa Java service
Documentation=https://zipkin.io/

[Service]
WorkingDirectory=/opt/zipkin
Environment="STORAGE_TYPE=cassandra3"
ExecStart=/usr/bin/java -jar zipkin.jar
User=zipkin
Group=zipkin
Type=simple
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

my zipkin-dependency.sh (crontab - every 5 mins)
STORAGE_TYPE=cassandra3 /usr/bin/java -jar /opt/zipkin/zipkin-dependencies.jar

crontab -l
*/5 * * * * cd /opt/zipkin && ./zipkin-dependencies.sh

any thoughts?

4 replies

Lukáš Vasek

@bilak

hello, please is there any support for azure's servicebus? I mean mainly this library. We are using two libraries in our spring boot services one is based on JMS and second one is based on mentioned library. While for JMS everything is provided by spring boot and brave/zipkin for servicebus library not. So I'd like to see what are the options and how difficult it would be to develop such library. Thannks

5 replies

Muchene

@Muchene

hi guys, would anyone here know why /api/v2/traces would return [] while /api/v2/trace/{trace_id} returns a valid trace?

1 reply

Archimedes Trajano

@trajano

Does Zipkin work Amazon keyspaces as the cassandra store?

Petr Hybler

@petr-hybler

hello guys, I have a question .... I have finally managed to make Zipkin + Cassandra work together... however, to properly display dependencies I need to periodically run zipkin-dependencies.jar (as mentioned above) ... I do this via cron in linux ... the problem is -> it takes a little over 1hr to calc the data...

Is there any way to limit the spans to automatically check for those nonprocessed only? or any other way I can tune cassandra / zipkin-dependencies.jar to make it work faster? Any tips?

I am not sure if I should focus on the cassandra tuning or the problem is somewhere else ... I tried to run twice manually the zipkin-dependencies and it says:
Running Dependencies job for 2021-03-01: 1614556800000000 =< Span.timestamp 1614643199999999 for both instances ... so it obivously recalc the same data ...

2 replies

deanlennox

@deanlennox

Hi,

I'm running zipkin in a container and sending a get (wrapFetch/node-fetch) trace to it (zipkin-transport-http):

const { Tracer } = require('zipkin');
const { BatchRecorder } = require('zipkin'); // format data spans
const { HttpLogger } = require('zipkin-transport-http');
const fetch = require('node-fetch');

const wrapFetch = require('zipkin-instrumentation-fetch');
const CLSContext = require('zipkin-context-cls'); // continous local storage
const ctxImpl = new CLSContext();

const localServiceName = 'xxx';

const recorder = new BatchRecorder({
logger: new HttpLogger({
endpoint: http://localhost:9411/api/v1/spans
})
});

const tracer = new Tracer({ ctxImpl, recorder, localServiceName });

const remoteServiceName = 'youtube';
const zipkinFetch = wrapFetch(fetch, { tracer, remoteServiceName });

zipkinFetch('https://www.youtube.com/').then(res => res)

How do I go about instrumenting and sending my own span?

Thanks,

3 replies

vadlamudi tejaswini

@vtejaswini_twitter

Is it the right practice for applications to wait for the Zipkin collector to be available? Can the application proceed further to serve its users and send no trace info as a collector is unavailable? Maybe the application can somehow automatically sense the collector and reload it-self. Am I thinking correct?

1 reply

Dhrumil0

@Dhrumil0

Hi everyone. My kafka cluster is using SASL_SSL GSSAPI. I have a kafka connect that is able to connect to authenticated cluster and produce and consume data from it. The issue is when I add the brave-kafka-interceptor and use the kafka collector, the connect is unable to produce data to cluster. The brave-kafka-interceptor is setup correctly because it works fine when using HTTP collector. I believe the interceptor also requires the SASL_SSL GSSAPI, truststore, keystore, jaas, krb configs when using with KAFKA collector. Does anyone know how I can set them? I already setup zipkin with the authentication configs and it seems to be connected to the cluster.

4 replies

Chris West

@FauxFaux

^ first time trying this thing out, presumably I'm missing something. My app talks to a db and a 3rd party service.
Seems like none of the folding or annotation stuff is available? I'm trying opentelemetry from node, there doesn't seem to be a way to override the serviceName for the http or the db? The remoteService? Is there some way to tag spans so they are remote? Change their kind? Really don't know where to look.

3 replies

Chris B

@afroDC

This may be answered somewhere else, but why does the Brave library expect headers to be in a case sensitive format when extracting them with an Extractor? For example it won't detect "x-b3-traceid" but will "X-B3-Traceid". I'm fairly certain the RFC for headers declares headers as case-insensitive

4 replies

Archimedes Trajano

@trajano

Is there an existing Apache HTTPD module that enables Zipkin tracing?

3 replies

Sathariel

@Sathariel

Hi all, running into a small issue connecting Zipkin to AWS Elasticsearch. Initially I was getting EmptyEndpointGroupException then I forced the health check off and realized that Zipkin couldn't find the AWS credentials so it's throwing "User: anonymous is not authorized to perform: es:ESHttpGet". This is what I'm using to execute:
STORAGE_TYPE=elasticsearch \
ES_HOSTS=https://<domain>.us-east-1.es.amazonaws.com/ \
ES_HEALTH_CHECK_ENABLED=false \
java -Dloader.path='aws.jar,aws.jar!/lib' -Dspring.profiles.active=aws \
-cp zipkin.jar org.springframework.boot.loader.PropertiesLauncher

I've even tried to forcing the environment variables but still same issue:
STORAGE_TYPE=elasticsearch \
ES_HOSTS=https://<domain>.us-east-1.es.amazonaws.com/ \
ES_HEALTH_CHECK_ENABLED=false \
AWS_ACCESS_KEY=<access> \
AWS_SECRET_KEY=<secret> \
java -Dloader.path='aws.jar,aws.jar!/lib' -Dspring.profiles.active=aws \
-cp zipkin.jar org.springframework.boot.loader.PropertiesLauncher

5 replies

Archimedes Trajano

@trajano

OK probably religious issue here.... which is the recommended one Cassandra or Elasticsearch for the Storage option? After mucking around with ES, it seems that unless it's managed by AWS or you dedicate a whole computer to it. ES is genreally a pain in the arse to work with. Mind you I think Cassandra looks to be a big beast to deal with as well.

1 reply

Archimedes Trajano

@trajano

I'm looking at https://zipkin.io/pages/architecture.html and I think there may need to be a second sequence diagram that shows two reporting components. From my current implementations (which could be wrong) it appears that each "service" must have access to the Zipkin reporter,

Arun Gopalpuri

@arun0009

How can we trace a call via reverse proxy using zipkin-go? Anyone has an example?

11 replies

sarkarsaurabh27

@sarkarsaurabh27

Hello folks, we are seeing some fatal error: concurrent map read and map write errors in our main application process and further debugging shows some issue with Zipkin library: github.com/openzipkin/zipkin-go

goroutine 182 [running]:
runtime.throw(0xe75af5, 0x21)
        /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc1c015f420 sp=0xc1c015f3f0 pc=0x4313f2
runtime.mapaccess2(0xcf7160, 0xc262f27e30, 0xc19f6391f0, 0xc19f6391f0, 0xc17eac231b)
        /usr/local/go/src/runtime/map.go:470 +0x278 fp=0xc1c015f468 sp=0xc1c015f420 pc=0x40fc58
reflect.mapaccess(0xcf7160, 0xc262f27e30, 0xc19f6391f0, 0xe6a8ed)
        /usr/local/go/src/runtime/map.go:1319 +0x3f fp=0xc1c015f4a0 sp=0xc1c015f468 pc=0x411f6f
reflect.Value.MapIndex(0xcf7160, 0xc1300544d0, 0x195, 0xcbb920, 0xc19f6391f0, 0x98, 0xcbb920, 0xc19f639250, 0x98)
        /usr/local/go/src/reflect/value.go:1176 +0x16d fp=0xc1c015f518 sp=0xc1c015f4a0 pc=0x49771d
. . .
github.com/openzipkin/zipkin-go/reporter.JSONSerializer.Serialize(...)
        /src/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.2.2/reporter/serializer.go:36
github.com/openzipkin/zipkin-go/reporter.(*JSONSerializer).Serialize(0x164b018, 0xc1951c4480, 0x64, 0x90, 0x0, 0xc0e90ac380, 0x433160, 0xea6f90, 0xc000056a00)
        <autogenerated>:1 +0x72 fp=0xc1c015fe78 sp=0xc1c015fe30 pc=0xb6f892
github.com/openzipkin/zipkin-go/reporter/http.(*httpReporter).sendBatch(0xc0000293b0, 0xc1c015ffaf, 0x1)
        /src/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.2.2/reporter/http/http.go:137 +0xb2 fp=0xc1c015ff90 sp=0xc1c015fe78 pc=0xb8a882
github.com/openzipkin/zipkin-go/reporter/http.(*httpReporter).sendLoop(0xc0000293b0)
        /src/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.2.2/reporter/http/http.go:99 +0x3f fp=0xc1c015ffd8 sp=0xc1c015ff90 pc=0xb8a49f
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc1c015ffe0 sp=0xc1c015ffd8 pc=0x4615c1
created by github.com/openzipkin/zipkin-go/reporter/http.NewReporter
        /src/go/pkg/mod/github.com/openzipkin/zipkin-go@v0.2.2/reporter/http/http.go:254 +0x2db

This is interesting as it is an error while serializing a Span for the Zipkin client. Upon closer inspection, there is a Tag map in the spans that seems a candidate for the concurrency issue. If the Span is being serialized and a new Tag or something is being added to it concurrently, we may run into an issue.

Does any of you have seen this issue or are aware of it? Any help would be appreciated.

Regards.

3 replies

Andreas Behr

@AndyBehr

Hi everyone, I think I am not seeing the obvious solution here. How do I configure how much storage Zipkin is allowed to use. I am using elasticsearch as a backend with the docker-compose files from Zipkin.

5 replies

Fran Pregernik

@FranPregernik

Hello!

I wanted to open a possible bug issue but came to ask here first.

I am using Spring Boot, Sleuth and Brave. There is a weird race condition that pops up and we are stumped.

The issue investigation is described here spring-cloud/spring-cloud-sleuth#1814 where Brave throws an exception "Parent can only be null in a local root". In the last message (https://github.com/spring-cloud/spring-cloud-sleuth/issues/1814#issuecomment-767389228) is the "stumped" part. In debug another subsequent call to the same method works fine.

Any help would be much appreciated.

6 replies

Somjit Nag

@Suedo

Is it possible to use custom id as a trace id in zipkin ?

https://stackoverflow.com/q/66777772/2715083

We have around 10 Spring boot microservices, which communicate with each other via kafka. At the start of any flow, a message-id is generated by one of our microservices, and that is propagated to all the others as part of the message transfer. We search for this message-id in our kibana logs to find the error.

So I wanted to understand if Zipkin can work with the message-id that our ecosystem already produces, instead of generating a new one ?

Ghost

@ghost~605bb9c76da0373984771c3a

B3 header question here. If I am packing up multiple traces to send to (in this case) datadog, do I just pick a trace to use for the b3 header or use multiple headers or some third thing?

2 replies

Petr Hybler

@petr-hybler

hello guys, I just have a question ... I have zipkin running together with cassandra as its persistent storage .. the issue I have is that with the external storage I have to recalc dependencies periodically .. it takes a bit over an hour to get that done ....

Is there the same issue with ElasticSearch ??? or is it faster? or do you have any tips how to speed it up??

tuhaihe

@tuhaihe

Hi there, I’m from StreamNative, now working at organizing the Pulsar Virtual Summit NA 2021. I’m trying to get in touch with our Zipkin community and invite as one community partner of Pulsar Summit NA 2021. In the past events, we have lots of successful experiences with different communities. Also, welcome community members to submit talks on Apache Pulsar + Zipkin(https://sessionize.com/pulsar-summit-north-america-2021).
I’m not sure this is the right place to talk about this. So, Looking forward to your reply. Thanks a lot!

1 reply

PapaWhiskey

@papawhiskey93

anyone help me for generating zipkin logs through kafka in c++ application

GesooBhatnagar

@GesooBhatnagar

Hi everyone, I am trying to install zipkin on my server an trying to use Go instrumentation. Is there a step by step guide to help me push the logs ?

4 replies

akash19980

@akash19980

I am using Zipkin with opentelemetry and now I want to use ElasticSearch as a storage for Zipkin. Can Anyone help me here

Fahri YARDIMCI

@ffahri

Hey, i have question regarding the OpenZipkin-Java (Brave) Tracing.currentTracer().currentSpan() usage. It returns lazy span. However when we try to finish that span using currentSpan().finish() it does not report the duration. Is this inteded ? I thought use the trace context in same thread different methods with currentSpan should work.

Denis Mikhaylov

@notxcain

Hi! We've upgraded to Finagle 21.2.0 and Scala 2.13 and now Finagle can't find HttpZipkinTracer even though it's definitely on the classpath. Any ideas why is this happenning? Thank you!

5 replies

Denis Mikhaylov

@notxcain

Hey folks! Any chance this PR get reviewed? openzipkin/zipkin-finagle#97

Iliadis Victor

@IliadisVictor

Hello guys i need info on brave testing techniques any feedback would be greatly appreciated

Parker Aldric Mar

@parker-mar

Hi, I am looking at the limit problem (openzipkin/zipkin#1142) and was wondering if this was solved completely? Thanks!

wmj765

@wmj765

Hi, everyone , how to understand the time gap between client start and server start ? In some of my traces , the time gap is even bigger than 11 seconds but all of requests are intranet communication .

[{"traceId":"607e3e086301ffaf74e217d5983efbc5","parentId":"74e217d5983efbc5","id":"91d3f96d11d7cb8e","kind":"SERVER","name":"get /api/msgcenter/msg/lastest","timestamp":1618886163861028,"duration":3775,"localEndpoint":{"serviceName":"base-msg-center","ipv4":"10.20.5.160"},"remoteEndpoint":{"ipv4":"123.157.238.102"},"tags":{"http.method":"GET","http.path":"/api/msgcenter/msg/lastest","mvc.controller.class":"ClientAPIController","mvc.controller.method":"getLastestMsgRecord"},"shared":true},{"traceId":"607e3e086301ffaf74e217d5983efbc5","parentId":"74e217d5983efbc5","id":"91d3f96d11d7cb8e","kind":"CLIENT","name":"get","timestamp":1618886152337857,"duration":11526274,"localEndpoint":{"serviceName":"lingyue-gateway","ipv4":"10.20.6.52"},"tags":{"http.method":"GET","http.path":"/api/msgcenter/msg/lastest"}},{"traceId":"607e3e086301ffaf74e217d5983efbc5","parentId":"74e217d5983efbc5","id":"864bdbd5a06a0515","kind":"SERVER","name":"get /auth/check","timestamp":1618886152333018,"duration":5224,"localEndpoint":{"serviceName":"dmo-user","ipv4":"10.20.7.38"},"remoteEndpoint":{"ipv4":"10.20.6.52","port":33926},"tags":{"http.method":"GET","http.path":"/auth/check","mvc.controller.class":"AuthController","mvc.controller.method":"checkToken"},"shared":true},{"traceId":"607e3e086301ffaf74e217d5983efbc5","id":"74e217d5983efbc5","localEndpoint":{"serviceName":"lingyue-gateway","ipv4":"10.20.6.52"},"annotations":[{"timestamp":1618886174659000,"value":"brave.flush"}]},{"traceId":"607e3e086301ffaf74e217d5983efbc5","parentId":"74e217d5983efbc5","id":"864bdbd5a06a0515","kind":"CLIENT","name":"get","timestamp":1618886152331773,"duration":5668,"localEndpoint":{"serviceName":"lingyue-gateway","ipv4":"10.20.6.52"},"tags":{"http.method":"GET","http.path":"/auth/check"}},{"traceId":"607e3e086301ffaf74e217d5983efbc5","id":"74e217d5983efbc5","kind":"SERVER","name":"get","timestamp":1618886152322007,"duration":11542205,"localEndpoint":{"serviceName":"lingyue-gateway","ipv4":"10.20.6.52"},"remoteEndpoint":{"ipv4":"123.157.238.102"},"tags":{"http.method":"GET","http.path":"/base-msg-center/api/msgcenter/msg/lastest"}}]

2 replies

Chandresh Pancholi

@chandresh-pancholi

Hi, How can i pass additional headers in zipkin-js. We have setup of microservices A --> B --> C. We want to pass header x-user-id from service A to service C. we are using zipkin-js.

Where communities thrive

People

Repo info

Activity