openzipkin/zipkin

Zipkin is a distributed tracing system

Adrian Cole

@adriancole

so you can confirm or deny this, by looking at the trace json.. don't even need to see logs to tell if something left unfinished was a problem

shengjiz

@shengjiz

Hello, I have a problem, there is one trace between two nodes : SpanA start in nodeA , SpanB start in nodeB after SpanA start , but the startTime of SpanB is smaller than startTime of SpanA in zipkin UI .

@adriancole I think the reason is that the clock between nodes is different, how can i resolve it ?

Adrian Cole

@adriancole

you can use something like NTP to keep your servers clocks in sync

Anders Clausen

@AndersClausen

Just out of curiosity - does Zipkin work with Active Directory? I'm thinking about how to access the dashboard in the cloud and one secure way would be for the users to log in with AD. Any ideas?

Chris K Wensel

@cwensel

sorry if this has been addressed before. but i’m seeing two SERVER spans for a given CLIENT span, making 3 spans, all with the same span ids. but the SERVER span meta-data is different. My guess is one of the SERVER spans is really a child span of the real SERVER span, but something went haywire. known issue?

sun12002

@sun12002

RabbitMQ collector for Zipkin doesn't have any option/configuration for setting a prefetch count on channel. What are the steps/guidelines for proposing a change?

Chris K Wensel

@cwensel

looking at one of the examples of this I found in our data, it feels like we are nesting CLIENT/SERVER spans like this: CLIENT0->CLIENT1->SERVER1->SERVER0. I say this because CLIENT0 is showing up as the parent to the other three spans, and it has no timestamp value.

Adrian Cole

@adriancole

@AndersClausen definitely sounds like a job for a proxy

@cwensel is the parentage CLIENT0->CLIENT1->SERVER1->SERVER0

or is it like CLIENT0->SERVER0, CLIENT0->CLIENT1->SERVER1

the latter is explainable when the async callback has the wrong span on it

which happens to be something tidying today

Chris K Wensel

@cwensel

@adriancole actually CLIENT2,SERVER1,SERVER0 all point to CLIENT0 as the parent, and CLIENT0 has no timestamp. if I sort the available timestamps, they order like CLIENT1->SERVER1->SERVER0, so I assumed CLIENT0 (with no timestamp) came first as its the parent of all of them.

Adrian Cole

@adriancole

well first no timestamp is a bug

no duration can be accidental, but no timestamp is a bug

server1 being a parent of server0 that sounds odd..

can you clean PII from the trace? and/or send to me

Chris K Wensel

@cwensel

client0 is the parent of everyone

(except itself)

Adrian Cole

@adriancole

root span?

or local root in that process

Chris K Wensel

@cwensel

there is, this is just a sub-graph of the trace

not sure about local root.. i’m just looking at raw data, i dont have the code available to me

(raw data before it gets to zipkin server)

Adrian Cole

@adriancole

I mean client0 shows parentID?

Chris K Wensel

@cwensel

to summarize, i have a sub-graph of 4 spans.. client0,client1,server0,server1. client0 is the parent to the others.. client0 has no timestamp. and client1,server1,server0 have timestamps in that order

Adrian Cole

@adriancole

can you turn client0,client1,server0,server1 into (span,localEndpoint) tuples for me?

trying to understand the remote relationships if any

so far I think it looks like a retry

though I can't explain client1, we could ignore it and then it looks like a retry

instrumentation should create a new kind=client (with a new span ID) for each try

but depending on how written, or limits of the framework, you can end up in this situation. in the UI and also the dep linker, we consider the localEndpoint to differentiate when the span IDs are the same

shengjiz

@shengjiz

I read the source code and found that microseconds are calculated by the local relative time(A trace on each node has a relative time), but the relative time of nodes is different, so whether the microseconds calculated at different nodes are comparable.

Adrian Cole

@adriancole

@shengjiz no they aren't strictly comparable. hierarchy can help determine which happened before the other in absolute time. in any case, keeping server times close is helpful

DDd

@webmutation

Hi, is there any up to date Zipkin Helm chart? I could only find the one from Financial Times, but that one seems outdated

I saw Jaeger helm in incubator and was surprised to not see one for zipkin

Adrian Cole

@adriancole

@webmutation dare you to make a pull request to the incubator with a zipkin one :D

DDd

@webmutation

if there is not one more up to date than the financial times one, I will look at their and see if there are many changes needed or not.

Javi

@Javigomez_gitlab

@adriancole

if a span was unfinished, there would be no duration for it

[
    [
        {
            "traceId": "0823811ca25bdad3",
            "parentId": "0823811ca25bdad3",
            "id": "6439eb5a54939542",
            "kind": "CLIENT",
            "name": "get",
            "timestamp": 1579883159955171,
            "duration": 809817,
            "localEndpoint": {
                "serviceName": "myservice"
            },
            "tags": {
                "http.path": "/user",
                "http.status_code": "200"
            }
        },
        {
            "traceId": "0823811ca25bdad3",
            "parentId": "0823811ca25bdad3",
            "id": "653ef3386c7c8139",
            "kind": "CLIENT",
            "name": "get",
            "timestamp": 1579883159955740,
            "duration": 817655,
            "localEndpoint": {
                "serviceName": "myservice"
            },
            "tags": {
                "http.path": "/u/settings",
                "http.status_code": "200"
            }
        },
        {
            "traceId": "0823811ca25bdad3",
            "parentId": "0823811ca25bdad3",
            "id": "58cdf6b54f3e50ae",
            "kind": "CLIENT",
            "name": "get",
            "timestamp": 1579883159965163,
            "duration": 819692,
            "localEndpoint": {
                "serviceName": "myservice"
            },
            "tags": {
                "http.path": "/u/accounts",
                "http.status_code": "200"
            }
        },
        {
            "traceId": "0823811ca25bdad3",
            "parentId": "0823811ca25bdad3",
            "id": "427b48ddce3ccacd",
            "kind": "CLIENT",
            "name": "get",
            "timestamp": 1579883159960778,
            "duration": 900222,
            "localEndpoint": {
                "serviceName": "myservice"
            },
            "tags": {
                "http.path": "/u/feature-flags",
                "http.status_code": "200"
            }
        },
        {
            "traceId": "0823811ca25bdad3",
            "parentId": "0823811ca25bdad3",
            "id": "147230185f20c741",
            "kind": "CLIENT",
            "name": "get",
            "timestamp": 1579883159953872,
            "duration": 1239779,
            "localEndpoint": {
                "serviceName": "myservice"
            },
            "tags": {
                "http.path": "/workers/mine",
                "http.status_code": "200"
            }
        },
        {
            "traceId": "0823811ca25bdad3",
            "parentId": "0823811ca25bdad3",
            "id": "07d4dbd088e6633c",
            "kind": "CLIENT",
            "name": "get",
            "timestamp": 1579883160880901,
            "duration": 552661,
            "localEndpoint": {
                "serviceName": "myservice"
            },
            "tags": {
                "http.path": "/workers/01D/feature",
                "http.status_code": "200"
            }
        },
        {
            "traceId": "0823811ca25bdad3",
            "parentId": "0823811ca25bdad3",
            "id": "b148e6b471cd2dfa",
            "kind": "CLIENT",
            "name": "get",
            "timestamp": 1579883160880282,
            "duration": 560505,
            "localEndpoint": {
                "serviceName": "myservice"
            },
            "tags": {
                "http.path": "/current-features",
                "http.status_code": "200"
            }
        },
        {
            "traceId": "0823811ca25bdad3",
            "parentId": "0823811ca25bdad3",
            "id": "bfc4a0a54601da77",
            "kind": "CLIENT",
            "name": "get",
            "timestamp": 1579883160886515,
            "duration": 562673,
            "localEndpoint": {
                "serviceName": "myservice"
            },
            "tags": {
                "http.path": "/workers/01D/feature",
                "http.status_code": "200"
            }
        },
        {
            "traceId": "0823811ca25bdad3",
            "

we can confirm that no spans are missing duration

Javi

@Javigomez_gitlab

How could we validate and prove that the problem is not in unfinished spans but that somehow we have "leaked context" problem?

Adrian Cole

@adriancole

@Javigomez_gitlab you can hack the recorder so that it not only reports spans but also logs them. in this case you could see if something seems fishy for example things ignoring their actual parent etc https://github.com/openzipkin/zipkin-js-example/blob/master/web/recorder.js#L24

Javi

@Javigomez_gitlab

thanks

Anders Clausen

@AndersClausen

@AndersClausen definitely sounds like a job for a proxy

Thanks @adriancole , I'll tell our platform team that :-D

javabotnetflix

@javabotnetflix

@AndersClausen which version of springboot you are using for your custom zipkin server? is that working for you?

Anders Clausen

@AndersClausen

@AndersClausen which version of springboot you are using for your custom zipkin server? is that working for you?

No it wasn't working for us, so we went ahead and used the 'ready-made' Zipkin server, which we are about to test very soon. Just setting up the CI/CD pipeline for it, so we can manage our Cassandra/Elasticsearch settings with it

Adrian Cole

@adriancole

@javabotnetflix are you a bot?

also do you accept movie requests?

I appreciate we have all the lethal weapon movies now, but...

Where communities thrive

People

Repo info

Activity