Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Rahul M Chheda
    @rahulchheda
    Congrats @tigrannajaryan ! That's great!
    Omer Katz
    @thedrow
    :tada:
    Tigran Najaryan
    @tigrannajaryan
    Thanks everyone for reviewing and providing valuable feedback. Next: logging libraries, log protocol and log data support in Collector.
    Reece Bradley
    @reecebradley
    ๐ŸŽˆ๐ŸŽ‰
    Tigran Najaryan
    @tigrannajaryan
    @/all I have to move the SIG meeting time today to 11am PT avoid the overlap with OpenTelemetry Community meeting.
    Pablo
    @voomik
    I am not sure if this is the correct location to ask this question; however, are there specific individuals that are allowed to participate in the Log SIG meetings? I am looking to listen on the calls, but I am open to contributing. TIA!
    Christian Beedgen
    @kumoroku
    calls are open, please do dial in!
    Pablo
    @voomik
    Fantastic. I will join :-)
    jeffalder
    @jeffalder
    Whatโ€™s the room for the 11am? Still https://zoom.us/j/8203130519 ?
    Pablo
    @voomik
    You were in the right room @jeffalder . Meeting ended already.
    Leighton Chen
    @lzchen
    Wasn't the meeting mved to 11am?
    Pablo
    @voomik
    Yes 11am PT
    Omer Katz
    @thedrow
    So how come it already ended?
    Or maybe I got the timezones confused
    Pablo
    @voomik
    Seems like the updates were light? That was my first time attending, so I am not familiar with the procedures.
    Christian Beedgen
    @raychaser
    yes, this was a very quick meeting today! we often go for the full hour
    generally, check out the meeting agenda/notes doc:
    https://docs.google.com/document/d/1cX5fWXyWqVVzYHSFUymYUfWxUK5hT97gc23w595LmdM โ€“โ€“read backwards a bit and it will hopefully provide a bunch of context
    as Tigran said, getting the Log Data Model approved was a big step that will now allow some implementation to start https://github.com/open-telemetry/oteps/blob/master/text/0097-log-data-model.md
    Omer Katz
    @thedrow
    But we need to decide how we want to do this
    Do any of you know eliot? The Python structured logging library?
    Tigran Najaryan
    @tigrannajaryan
    @/all please review a tiny amendment to Log Data Model: open-telemetry/oteps#109
    Omer Katz
    @thedrow

    Do any of you know eliot? The Python structured logging library?

    I'd like to start discussing the API for our logging SDK

    Anyone up for it?
    Avi Kessner
    @avik-so
    I'm starting to define an internal project for logging, and I was hoping to be able to take advantage of open telemetry. I'd be happy to participate in the logging API.
    Omer Katz
    @thedrow
    @avik-so That's great.
    Anyone else?
    28 replies
    Tristan Sloughter
    @tsloughter
    would opentelemetry logs encompass error/exception reporting? I feel like it is the same data just maybe represented differently
    4 replies
    Christian Beedgen
    @kumoroku
    regarding defining a logging API: when the SIG started in late March, this was discussed. see for example the notes for March 25 and April 8โ€“โ€“the group at that point decided to focus on integrating with existing logging libraries first, then tackle the logging API. the overall sense was that most projects already had integrated with a library (log4j, ...) and forcing the use of yet something else was going to be an uphill battle.
    Omer Katz
    @thedrow
    I somewhat agree
    But new projects would probably use our API
    Christian Beedgen
    @kumoroku
    I think this is sound logic, but I will also say that personally, I'd love to see OT as a one-stop shop for all telemetry instrumentation, and as you are saying, what about the next 100,000 projects? :)
    personally, I think if there's critical mass around defining a logging API among folks participating here, nobody should keep anybody from making progress there, and I do not think the original prioritization was meant to prevent this. it was just that the folks being able to spend time on logging within OT in general at that time wanted to dig into integrating with existing first
    I think it is a fine topic to discuss at the SIG meeting
    Omer Katz
    @thedrow
    I agree
    Joshua MacDonald
    @jmacd
    I think everyone likes the idea of a logging API, but the people who've been working on this project since 2019 are more interested in finishing the tracing and metrics APIs. I don't think you'll get full attention on logging APIs until then. (P.S. I've paid zero attention to the logging SIG because I'm in this boat, I've got a metrics API to finish--and yet I've designed and implemented numerous logging APIs in C++ and Go.) I'd be happy if all the people interested in logging would pitch in on tracing and metrics!
    Steve Flanders
    @flands
    2019 member here โ€” I do not think it is a reasonable expectation that everyone be interested in every aspect of the project. Plenty of people only care about a subset of observability data. Also, as the community grows we should leverage all the skillsets available โ€” we have many new additions who are primarily interested in logs including AWS, Elastic, and SumoLogic. While it would be great to be involved in every aspect that is not feasible as the project grows. Of course we should not slow down on the aspects on the path to GA being traces and metrics and @mtwo already covered the goal is to GA later this year with logs likely entering beta.
    10 replies
    Joshua MacDonald
    @jmacd
    HI all, I've been thinking about how this Logging sub-project could help more with tracing and metrics. I was thinking that before beginning to work on Logging API design, we could try to standardize the translation from Span and Metric events into structured logging events. This was requested as a feature for the metric SDK (open-telemetry/opentelemetry-specification#617), and I think the same could be said of spans. Right now many OTel repos are including standard-output exporters for both trace and metrics, but there are no standard conventional data formats for them to use, so it's a bit arbitrary what you get. I think it would be great if we had a standard structured log form for both metrics and spans.
    I would add that the OTel metrics API was conceived as a logging API from the start. The data model describes metric events, which can be recorded as a faithful way to capture metrics, so we would just need to standardize how they are structured as log records. Here's the event structure: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/metrics/api.md#metric-event-format
    Tigran Najaryan
    @tigrannajaryan
    @jmacd do you mean a standardized way to encapsulate span data and metric data inside a log record or something else?
    Joshua MacDonald
    @jmacd
    yes
    say you wanted to simply write spans into elastic search, for example
    Tigran Najaryan
    @tigrannajaryan
    Got it. Do you want to do an OTEP? (Assuming you want to get distracted from metrics :-) )
    Joshua MacDonald
    @jmacd
    I don't. I can't :)
    Ryan-Git
    @Ryan-Git
    Hi all. I just come up with an idea. As we will transfer logs to a remote collector in some sense, could we leverage dictionary encoding? Just like nano log(https://www.usenix.org/conference/atc18/presentation/yang-stephen). Briefly it makes a dictionary for static parts of every log message and writes only dynamic part as a compact binary log at runtime, which is super fast. This binary form is easier to transfer and analyze. A post processor is needed to make it human readable.
    8 replies
    Omer Katz
    @thedrow
    @jmacd I think you're on the right path
    51 replies
    Mats Taraldsvik
    @meastp

    Hi - I'd sometimes like to upload data dumps / full request dumps etc. and couple this to a specific trace (event/attribute). I think this might belong in the logging camp. However, I beleive that the data dumps/blobs are too big to transport to the collector via exporters (??) What I was thinking, is to use a logging attribute e.g. 'dump.file_path' in the application and write the file to /tmp, and in the agent (collector on the same node) in a "custom" processor:

    1. look for 'dump.file_path'
      2 if the file exists:
      3 upload to a cloud service (e.g. Azure Blob Storage)
    2. replace 'dump.file_path' with 'dump.file_uri' from the cloud service

    Is this a sensible thing to do?

    3 replies
    David Poncelow
    @zenmoto
    Hi- I've got an OTEP PR going about adding correlation information to existing logs at https://github.com/open-telemetry/oteps/pull/114- I'd appreciate any feedback about it or approvals if it looks good.
    6 replies
    Tigran Najaryan
    @tigrannajaryan
    @/all please review Logs protocol PR: open-telemetry/opentelemetry-proto#151
    Avi Kessner
    @avik-so
    I'm working on the OTEP for storing metric and tracing data into the log format.
    Which formats should I be looking at besides OT, Zipkin and Jaeger?
    7 replies