We are starting on our magical journey for custom traces.. (we are using opencensus to actually do the tracing but the gitter seems pretty dead). We have deployed the opencensus agent, opencensus collector which is writing to jaeger-collector and then into the jaeger stack. We are sending traces from 2 sources.
I have some questions regarding trace sampling ratios, settings and in which places.. Right now we are using the default sampling probability everywhere, but it doesnt work so well. We basically end up with traces from the two diffrent sources being disjointed from eachother.. Our applications check for the root span id, if its there then they use it (its always there because istio generates it) otherwise it would create a new one.
Should the apps + istio have a tracing probability of 100% and then let the opencensus collector control the probability and produce more accurate spans? Or? Tracing every request is probally not the best idea due to the overhead it creates.