Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Adrian Cole
@adriancole
you are pasting a timeout, but the hostname/port of elasticsearch is default (the not https one)
Are you trying to troubleshoot inability to connect to elasticsearch? If so, you can set ES_HTTP_LOGGING=BASIC for example
also /prometheus will have a lot of stats about the client
but I don't understand the relationship between TLS/https and the stack trace you mention because the port is the plain text one (9200)
Adrian Cole
@adriancole
also you get massive stats like this, too curl -sSL localhost:9411/prometheus|grep 'elastic.*0'
there are deeper logging options available, but I would first check the node your server is on can reach 10.0.88.33:9200 using http protocol
Luis Weir
@luisw19

@adriancole I used zipkin-js-example as input to debug my implementation of zipkin-js and tha's how I was able to generate the logs.. In my case I have a Node service that uses express (called service in the earlier log) instrumented pretty much as zipkin-js-example, however the service makes calls to other backend services (target-a and target-b in the previous log) using zipkin-instrumentation-request-promise.

Helicopter view of the service flow is: call -> express -> route controller (singleton) -> invoker (singleton). Isinvoker that implements zipkin-instrumentation-request-promise pretty much as described in Using the Request class section of https://github.com/openzipkin/zipkin-js/tree/master/packages/zipkin-instrumentation-request-section.

The issue described appears when a given call takes longer to be processed (by the controller) than a newer call. In the logs I shared the two calls are represented with two distinct traceIds:1000000000000001 and 1000000000000002.

Adrian Cole
@adriancole
@luisw19 my suspicion is that you would have the same behavior if two calls were made to the same endpoint vs different hosts
Stanleytam
@stanltam_twitter
image.png
Luis Weir
@luisw19
@adriancole you mean two backend endpoints?
Adrian Cole
@adriancole
no exactly same
you just call it twice in the same pattern
because I think this hasn't to do with the endpoints being on different hosts
rather something about how the propagation is going
Luis Weir
@luisw19
@adriancole yes, I agree, I don't think the issue is to do with the fact am calling two different target services.
Adrian Cole
@adriancole
basically I'm challenging you to reduce variables until it reproduces issue with least new code
then you might be able to get a branch off normal example with only several lines changed in other words
easier to solve
@stanltam_twitter looking at image now..
so you are literally listening on ssl to the same port as not ssl?
Stanleytam
@stanltam_twitter
Actually I have already set to https... and the elastic is using 9200 for https port. I am able to use logstash to connect with port 9200...
Seems prometheus API don't return much hints...
elasticsearch_healthcheck_responseLength{httpStatus="400",method="GET",quantile="0.0",} 256.0
elasticsearch_healthcheck_responseLength{httpStatus="400",method="GET",quantile="0.5",} 256.0
elasticsearch_healthcheck_responseLength{httpStatus="400",method="GET",quantile="0.75",} 256.0
elasticsearch_healthcheck_responseLength{httpStatus="400",method="GET",quantile="0.9",} 256.0
elasticsearch_healthcheck_responseLength{httpStatus="400",method="GET",quantile="0.95",} 256.0
elasticsearch_healthcheck_responseLength{httpStatus="400",method="GET",quantile="0.98",} 256.0
elasticsearch_healthcheck_responseLength{httpStatus="400",method="GET",quantile="0.99",} 256.0
elasticsearch_healthcheck_responseLength{httpStatus="400",method="GET",quantile="0.999",} 256.0
elasticsearch_healthcheck_responseLength{httpStatus="400",method="GET",quantile="1.0",} 256.0
elasticsearch_healthcheck_responseLength_count{httpStatus="400",method="GET",} 4.0
elasticsearch_healthcheck_responseLength_sum{httpStatus="400",method="GET",} 1036.0
elasticsearch_healthcheck_responseLength_max{httpStatus="400",method="GET",} 259.0
elasticsearch_healthcheck_timeouts_total{cause="ResponseTimeoutException",httpStatus="400",method="GET",} 0.0
elasticsearch_healthcheck_timeouts_total{cause="WriteTimeoutException",httpStatus="400",method="GET",} 0.0
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds{httpStatus="400",method="GET",quantile="0.0",} 0.006291456
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds{httpStatus="400",method="GET",quantile="0.5",} 0.007077888
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds{httpStatus="400",method="GET",quantile="0.75",} 0.007077888
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds{httpStatus="400",method="GET",quantile="0.9",} 0.352059392
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds{httpStatus="400",method="GET",quantile="0.95",} 0.352059392
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds{httpStatus="400",method="GET",quantile="0.98",} 0.352059392
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds{httpStatus="400",method="GET",quantile="0.99",} 0.352059392
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds{httpStatus="400",method="GET",quantile="0.999",} 0.352059392
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds{httpStatus="400",method="GET",quantile="1.0",} 0.352059392
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds_count{httpStatus="400",method="GET",} 4.0
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds_sum{httpStatus="400",method="GET",} 0.356786771
elasticsearch_healthcheck_connectionAcquisitionDuration_seconds_max{httpStatus="400",method="GET",} 0.336022085
elasticsearch_healthcheck_totalDuration_seconds{httpStatus="400",method="GET",quantile="0.0",} 0.001376256
elasticsearch_healthcheck_totalDuration_seconds{httpStatus="400",method="GET",quantile="0.5",} 0.001769472
elasticsearch_healthcheck_totalDuration_seconds{httpStatus="400",method="GET",quantile="0.75",} 0.001769472
elasticsearch_healthcheck_totalDuration_seconds{httpStatus="400",method="GET",quantile="0.9",} 0.010420224
elasticsearch_healthcheck_totalDuration_seconds{httpStatus="400",method="GET",quantile="0.95",} 0.010420224
elasticsearch_healthcheck_totalDuration_seconds{httpStatus="400",method="GET",quantile="0.98",} 0.010420224
elasticsearch_healthcheck_totalDuration_seconds{httpStatus="400",method="GET",quantile="0.99",} 0.010420224
elasticsearch_healthcheck_totalDuration_seconds{httpStatus="400",method="GET",quantile="0.999",} 0.010420224
elasticsearch_healthcheck_totalDuration_seconds{httpStatus="400",method="GET",quantile="1.0",} 0.010420224
elasticsearch_healthcheck_totalDuration_seconds_count{httpStatus="400",method="GET",} 4.0
elasticsearch_healthcheck_totalDuration_seconds_sum{httpStatus="400",method="GET",}
Adrian Cole
@adriancole
protip wrap in ``` otherwise it looks like a hundred comments
ok so you are asking how to change the TLS setup used by the client, not the server listener
I think before I misunderstood or maybe two people ask different tls questions at the same time :P
Stanleytam
@stanltam_twitter
haha i see... yes, the target elastic is protected by cert auth.... And I try to connect zipkin with that ...But not success...
Adrian Cole
@adriancole
ok I will hold the patch release we are about to do until figure out the answer
first let me check last TLS answer :
Luis Weir
@luisw19
@adriancole sure I'll try and reproduce reducing as many variables as possible (e.g. same target sevice, coupling controller and invoker in the same class). I'll let you know how it goes.
Adrian Cole
@adriancole
as few luis.. as few! :D
Luis Weir
@luisw19
lol
Adrian Cole
@adriancole
@mvbjoshi seems default spring boot properties for ssl should already work, can you just try it please? line/armeria#1663
@stanltam_twitter that answer isn't for you heh
lemme first get the server thing going.. changing tests
as this is spring boot you can choose whether you want to do that with --x.y.z=value or an additional properties file, or something else
@stanltam_twitter openzipkin/zipkin#2774
usually response time is less than a day, so happy friday :P
adriancole @adriancole wanders away but will check back in a couple answers in case magic happens
Adrian Cole
@adriancole
@stanltam_twitter those settings you mentioned.. that's it right?
keystore, truststore and passwords
Stanleytam
@stanltam_twitter
yup:D, thanks bro
Adrian Cole
@adriancole
@stanltam_twitter try? openzipkin/zipkin#2775
hoping it works. I'll check up in the morning
gnight
Stanleytam
@stanltam_twitter
@adriancole , hey bros, you are my hero!!! My zipkin finally can connect to the elasticsearch . Thank you so much, you are so efficient, looking forward for the official version haha
bhaveshhpatel
@bhaveshhpatel
@adriancole Thanks for your reply. So reading the headers and stuffing in logging (state being parsed from the header and stuffed into TraceContext) is what I was looking for. Any examples I can look at? I have looked at the ScopeDecorator. Thanks in advance.
can't find any similar examples in the zipkin github repo
Adrian Cole
@adriancole
@bhaveshhpatel if you are using sleuth, it already does this sort of thing. can you rephrase what you need? maybe already covered? https://cloud.spring.io/spring-cloud-sleuth/reference/html/#_propagating_extra_fields