Hey all, just REALLY looking so hard for info on this:
I have ingress-nginx ingress on GKE. I configured the ingress with a Zipkin tracer that sends to Stackdriver. with enable-open-tracing. Then i also installed the google stackdriver tracing library in my c# api to send spans to stackdriver. But it looks like in the api it is very difficult to figure out how to CONTINUE the span FROM the trace started in the Ingress. So in Stackdriver I always start a new trace when the controller in my api is hit. as opposed to continuing from the ingress.
i want it to be ONE trace, with a span for the ingress, and a span for the api
I've so far not seen anything related to the code we maintain in your description, except for the word zipkin
this channel is volunteer run, so it is best to get you where you need to go. understand troubleshooting things can be frustrating, but not sure how much we can help on something we have no responsbility over either
so first what is "Zipkin tracer that sends to Stackdriver" this will start the triage
many things call themselves zipkin, and when you said that option seems like it isn't likely something we wrote
that's what i'm trying to do.
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-opentracing
That shows how ingress-nginx can send to a zipkin host.
I have a
From this, being that it is sending the logs to stackdriver, I can see a trace from ingress-nginx. that request went to my c# api.
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-opentracing
That shows how ingress-nginx can send to a zipkin host.
I have a
docker.io/openzipkin/zipkin-gcp container in my cluster.From this, being that it is sending the logs to stackdriver, I can see a trace from ingress-nginx. that request went to my c# api.
you've helped me with this container in the past... either way, for now, I'm just trying to understand if usually people connect all the spans under one trace, or not necessarily
because any tracing library i try to use in code, doesn't pick up the trace that already started in the ingress and simply add to it with spans. It starts a new trace
so either the ingress-nginx isn't sending b3 headers, or whatever your c# api is isn't reading them
based on your description of the problem anyway (start a new trace problem)
so it is important to know what c# thing you are using, which headers nginx are sending and if that thing supports reading those or not
ex there are two forms of B3 headers (multi and single header) https://github.com/openzipkin/b3-propagation
to triage you can start with this.. if the zipkin-gcp thing is receiving spans, that's good. it cannot affect propagation though
that will be the seam between your apps in this case nginx and the next thing
@nicmunroe has a nice piece on the general art of troubleshooting which applies regardless of library type https://medium.com/nikeengineering/hit-the-ground-running-with-distributed-tracing-core-concepts-ff5ad47c7058
and to answer directly, yes it is the intent to connect all spans in one http request together with the same trace ID
adriancole @adriancole has to go for a bit
thank you for all the info. I have the answers i need. the request after it goes through the nginx ingress has b-3 headers. Maybe i will trouble shoot more within the zipkin library in c#. (which is https://github.com/openzipkin/zipkin4net)
continuing the docker/zipkin/eureka saga I switched to the other Dockerfile strategy:
# We copy files from the context into a scratch container first to avoid a problem where docker and
# docker-compose don't share layer hashes https://github.com/docker/compose/issues/883 normally.
# COPY --from= works around the issue.
FROM scratch as scratch
COPY . /code/
FROM openzipkin/zipkin-builder as built
COPY --from=scratch /code /code
WORKDIR /code
RUN mvn -B --no-transfer-progress package -DskipTests=true
WORKDIR /zipkin-eureka
RUN cp /code/target/eureka-zipkin-client-*-module.jar eureka-zipkin-client-module.jar && \
(mkdir eureka-zipkin-client-module && cd eureka-zipkin-client-module && jar -xf ../eureka-zipkin-client-module.jar) && \
rm eureka-zipkin-client-module.jar
RUN cp /code/eureka-client.properties eureka-client.properties
FROM openzipkin/zipkin:master
MAINTAINER Zipkin "https://zipkin.io/"
COPY --from=built /zipkin-eureka/ /zipkin/
ENV MODULE_OPTS="-Dloader.path=eureka-zipkin-client-module -Dlogging.level.*=DEBUG -Dlogging.level.org=DEBUG -Dlogging.level.com=DEBUG -Dlogging.level.io=DEBUG -Dspring.application.name=zipkin"As was suggested by @adriancole further up in the channel (Thank you!) and with that I upped the logger to DEBUG. That gave this helpful info on what spring was loading:
============================
CONDITIONS EVALUATION REPORT
============================
Positive matches:
-----------------
....
Negative matches:
-----------------
...
EurekaDiscoveryClientConfigServiceBootstrapConfiguration:
Did not match:
- @ConditionalOnClass did not find required class 'org.springframework.cloud.config.client.ConfigServicePropertySourceLocator' (OnClassCondition)
...I tried also adding spring-cloud-starter-config-client, but that gave issues with missing dependencies. Unfortunately I'm new to spring boot (just started playing with all this yesterday) so I'm not sure where to go from here.
Currently I have eureka setup for service discover in spring boot mostly because that was the first tutorial I was looking at... would Consul be easier to setup with the zipkin docker image? Up until this Eureka has been easy to setup for the other services I was working with, but I haven't tried consul.
what you probably have not tuned right is the jar added to the loader path. it likely is missing an inner thing including the cloud-config library
the "module" jar should have packed that up in other words
that all said, yeah anything that doesn't require modifications to binaries is easier
@thpeng if you are using kafka, sure it has some natively supported stuff, as does rabbitmq I think. http we didn't choose to bake in something as there are major issues with basic auth, credential rotation etc we didn't want to step into. Instead just use a proxy that has the features you need
you can call it mesh :)
@adriancole thanks. We're stuck with the rest endpoint because of private PaaS setup reasons. I thought about moving zipkin behind the spring-cloud-gateway, securing all non-GET endpoints (allowing the usage of the ui freely). As far as I have seen, the zipkin ui does not use other methods (upload excluded). But then we have also to modify the span reporter in sleuth (using a customized rest template).
the upload is actually in-browser.. doesn't get to the server
so the POST thing is indeed collector-only (also under /api/v2/spans, not /zipkin) and that narrows the thing to pin
ex some SaaS offerings run zipkin clones and have different auth mecahnisms
good luck!
I have configured brave for trace id and span id in spring web-mvc project. It is showing the server call but it is not showing the call which I am making from spring mvc project to spring boot project because of which trace id is different in spring MVC and spring boot. Please advise.
There should be two span for one trace Id.. one will refer to the server call and another to the client call