Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Oct 13 19:37
    seglo commented #1061
  • Oct 13 19:31
    seglo edited #1061
  • Oct 13 19:19
    seglo opened #1061
  • Oct 12 22:52
    Voltir edited #1060
  • Oct 12 22:51
    Voltir opened #1060
  • Oct 08 15:16
    ivantopo synchronize #1057
  • Oct 08 15:16

    ivantopo on scala3-initial-support

    increase timeout for setup quer… (compare)

  • Oct 08 14:51

    ivantopo on scala3-initial-support

    increase wait time for Cassandr… (compare)

  • Oct 08 14:50
    ivantopo synchronize #1057
  • Oct 08 13:27
    ivantopo synchronize #1057
  • Oct 08 13:27

    ivantopo on scala3-initial-support

    try running on self-hosted acti… (compare)

  • Oct 08 12:52
    ivantopo synchronize #1057
  • Oct 08 12:52

    ivantopo on scala3-initial-support

    make the reporters cross-build … (compare)

  • Oct 08 11:18
    ivantopo synchronize #1057
  • Oct 08 11:18

    ivantopo on scala3-initial-support

    forward beforeAll and afterAll … (compare)

  • Oct 08 10:01
    ivantopo synchronize #1057
  • Oct 08 10:01

    ivantopo on scala3-initial-support

    add InitAndStopKamonAfterAll to… (compare)

  • Oct 08 09:41
    ivantopo synchronize #1057
  • Oct 08 09:41

    ivantopo on scala3-initial-support

    migrate SBT settings to slash s… try printing found Spans on the… (compare)

  • Oct 08 08:34
    ivantopo synchronize #1057
Rajat Khandelwal
@prongs

In my application, there is a long living WebSocket for a user session. The user interacts on the UI, which sends messages on the WebSocket, and the application does a bunch of things and sends one or multiple messages back to the UI. I'm unable to get traces for this in jaeger. I'm able to get traces for other HTTP requests, but not WebSocket requests. Is there any examples around this that I could follow? I'm on play 2.6 and using WebSocket like https://www.playframework.com/documentation/2.6.x/ScalaWebSockets. I've tried both Netty Server and Akka Server behind the scenes, but the result is same: HTTP requests are getting traced, websocket is not.

Ideally I'd like the whole user session traced, with spans for messages and other sub-operations like db calls, outbound HTTP calls etc.

1 reply
Ivan Topolnjak
@ivantopo
as far as I know, nope
no examples that we could share, but maybe someone else around had experience with this?
Rajat Khandelwal
@prongs
Can I create a long living kamon context and use that for processing all messages on that websocket? I tried that as well, didn't seem to work.
Ivan Topolnjak
@ivantopo
in this case I'm thinking that you will have to start a new trace on each message that comes on the websocket and then let the rest of Kamon's instrumentation create the spans for database/http and other stuff
^^ it depends
do you have any pieces of information from the initial setup that you always want to keep as part of the context when processing all other requests coming through the socket?
Rajat Khandelwal
@prongs
yeah, let's say I have user id and session id
Ivan Topolnjak
@ivantopo
and, do you have any super tiny test example that we could use to attempt some instrumentation on?
Rajat Khandelwal
@prongs
difficult to break this, as I'm currently testing with UI, and that makes both HTTP calls and WebSocket calls. Could be a bunch of effort to break this down to a super tiny example :|
Ivan Topolnjak
@ivantopo
mmmm
maybe some of the public play examples?
1 reply
I'm asking because we don't use websockets ourselves and I don't have any pet project we could use to try this out
but if there is anything out there that could be close enough to what you are building we could give it a try!
Rajat Khandelwal
@prongs
sure, I can try with a public project, could take some time though. Meanwhile, If I were to retry the long-living-context thing, do you think it should work out of the box?
Ivan Topolnjak
@ivantopo
not out of the box
I think that with a bit of manual instrumentation you should be able to keep the user/session ids in a context and then create a new context with a new Span for each message that comes through that socket
out for lunch, will be back in about an hour!
Ivan Topolnjak
@ivantopo
back
Rajat Khandelwal
@prongs
My websocket actor will be doing some future operations and will send some messages to other actors. If I wrap these operations with Kamon.runWithContext(mySavedContext), then it should work to some extent, right?
Ivan Topolnjak
@ivantopo

sort of yes.. let me elaborate: if that initial Context has a not-sampled Span in it and you reuse that same Context then all other Spans that would be created by those actor messages will not be sampled as well. Also, all of the Spans would be part of the same trace which most likely will turn the trace useless if you end up having thousands and thousands of Spans spanning (possibly) hours.

I would recommend you manually create a Context with the user/session ids when the actor handling the websocket is created and then, every time you get a new "command" into that actor you create a new span for that new command, call yourOriginalContext.withEntry(Span.Key, newSpanForThisOperation) and then use Kamon.runWithContext

Rajat Khandelwal
@prongs
sure, let me try that. Can I still keep the global span (for the entire WebSocket) and manually close that on actor destroy, so that there is one span for the whole session?
Ivan Topolnjak
@ivantopo
Yeap, there is no problem with that!
Rajat Khandelwal
@prongs
alright, so I'm trying out something like this:
    case w: WebSocketMessage ⇒
      Kamon.runWithSpan(Kamon.spanBuilder(w.msgType.toString).start()) {
        handleWebSocketMessage(arg1, arg2, arg3)(w)
      }
Rajat Khandelwal
@prongs
I'm able to get some of the messages as individual traces. How do I tie them together in one trace?
and I'm getting individual traces for my outbound HTTP calls, db calls etc.
Ivan Topolnjak
@ivantopo
so, for example, one incoming message in the socket generates a Span and a related JDBC/HTTP call also generates a Span but those Spans are not tied on the same trace?
Rajat Khandelwal
@prongs
yup
Ivan Topolnjak
@ivantopo
do you have any Cats or Monix in between the websocket and the JDBC/HTTP calls?
Rajat Khandelwal
@prongs
I think context is not being propagated somehow with both Kamon.runWithContext and Kamon.runWithSpan
no cats or monix
(akka + play + slick)
Ivan Topolnjak
@ivantopo
ok.. what versions are you using?
of Play, Slick and Kamon
Rajat Khandelwal
@prongs
  val akkaVersion           = "2.6.5"
  val playVersion           = "2.6.13"
  val playSlickVersion      = "3.0.3"
  val kamonVersion          = "2.1.1"
Ivan Topolnjak
@ivantopo
and also using the SBT plugin for dev mode?
Rajat Khandelwal
@prongs
addSbtPlugin("io.kamon" % "sbt-kanela-runner-play-2.6" % "2.0.6")
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.9.2")
resolvers += Resolver.bintrayIvyRepo("kamon-io", "sbt-plugins")
addSbtPlugin("io.kamon" % "sbt-aspectj-runner-play-2.6" % "1.1.2")
but I'm not running in dev mode as of now. It's packaged as docker image and running in k8s.
Ivan Topolnjak
@ivantopo
ok ok
are you able to access the status page on that container and see what's the status of the instrumentation modules?
Rajat Khandelwal
@prongs
Actually I was targeting 3 of our services with Kamon. 2 of them are working great. The 3rd one had WebSockets. In that also, HTTP is working fine, only WebSocket is problematic. I've been debugging that since 2 days now, tried doing create context at different places (mostly wrapping the message passing to worker actors), but looks like somewhere it gets lost or something.
curl for instrumentation module gives this json
{
  "present": true,
  "modules": {
    "annotation": {
      "name": "Annotation Instrumentation",
      "description": "Provides a set of annotations to create Spans and Metrics out of annotated methods",
      "enabled": true,
      "active": false
    },
    "akka-http": {
      "name": "Akka HTTP Instrumentation",
      "description": "Provides context propagation, distributed tracing and HTTP client and server metrics for Akka HTTP",
      "enabled": true,
      "active": true
    },
    "executor-service": {
      "name": "Executor Service Instrumentation",
      "description": "Provides automatic Context propagation to all non-JDK Runnable and Callable implementations which enables\n         Context propagation on serveral situations, including Scala, Twitter and Scalaz Futures",
      "enabled": true,
      "active": true
    },
    "play-framework": {
      "name": "Play Framework Instrumentation",
      "description": "Provides context propagation, distributed tracing and HTTP client and server metrics for Play Framework",
      "enabled": true,
      "active": true
    },
    "mongo-driver": {
      "name": "Mongo Driver Instrumentation",
      "description": "Provides automatic tracing of client operations on the official Mongo driver",
      "enabled": true,
      "active": false
    },
    "akka-remote": {
      "name": "Akka Remote Instrumentation",
      "description": "Provides distributed Context propagation and Cluster Metrics for Akka",
      "enabled": true,
      "active": true
    },
    "jdbc": {
      "name": "JDBC Instrumentation",
      "description": "Provides instrumentation for JDBC statements, Slick AsyncExecutor and the Hikari connection pool",
      "enabled": true,
      "active": false
    },
    "scala-future": {
      "name": "Scala Future Intrumentation",
      "description": "Provides automatic context propagation to the thread executing a Scala Future's body and callbacks",
      "enabled": true,
      "active": true
    },
    "akka": {
      "name": "Akka Instrumentation",
      "description": "Provides metrics and message tracing for Akka Actor Systems, Actors, Routers and Dispatchers",
      "enabled": true,
      "active": true
    },
    "logback": {
      "name": "Logback Instrumentation",
      "description": "Provides context propagation to the MDC and on AsyncAppenders",
      "enabled": true,
      "active": true
    }
  },
  "errors": {}
}
Ivan Topolnjak
@ivantopo
when it gets to this point and you need to start debugging it gets a bit annoying but relatively easy: println eveywhere! :joy:
I would start logging the current trace id everywhere and see where it gets lost
if it is on a future, actor or something we already support then maybe there is an issue with the agent or initialization... if it is on something else that we don't support at the moment then new (or manual) instrumentation will be necessary
can you please try to narrow it down the specific interaction where the context is lost? and of course, ask if you need any help!
Rajat Khandelwal
@prongs
appreciate the help :)
Ivan Topolnjak
@ivantopo
if there are any akka streams in the middle that could also be breaking propagation