by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Omer Katz
    @thedrow
    I somewhat agree
    But new projects would probably use our API
    Christian Beedgen
    @kumoroku
    I think this is sound logic, but I will also say that personally, I'd love to see OT as a one-stop shop for all telemetry instrumentation, and as you are saying, what about the next 100,000 projects? :)
    personally, I think if there's critical mass around defining a logging API among folks participating here, nobody should keep anybody from making progress there, and I do not think the original prioritization was meant to prevent this. it was just that the folks being able to spend time on logging within OT in general at that time wanted to dig into integrating with existing first
    I think it is a fine topic to discuss at the SIG meeting
    Omer Katz
    @thedrow
    I agree
    Joshua MacDonald
    @jmacd
    I think everyone likes the idea of a logging API, but the people who've been working on this project since 2019 are more interested in finishing the tracing and metrics APIs. I don't think you'll get full attention on logging APIs until then. (P.S. I've paid zero attention to the logging SIG because I'm in this boat, I've got a metrics API to finish--and yet I've designed and implemented numerous logging APIs in C++ and Go.) I'd be happy if all the people interested in logging would pitch in on tracing and metrics!
    Steve Flanders
    @flands
    2019 member here — I do not think it is a reasonable expectation that everyone be interested in every aspect of the project. Plenty of people only care about a subset of observability data. Also, as the community grows we should leverage all the skillsets available — we have many new additions who are primarily interested in logs including AWS, Elastic, and SumoLogic. While it would be great to be involved in every aspect that is not feasible as the project grows. Of course we should not slow down on the aspects on the path to GA being traces and metrics and @mtwo already covered the goal is to GA later this year with logs likely entering beta.
    10 replies
    Joshua MacDonald
    @jmacd
    HI all, I've been thinking about how this Logging sub-project could help more with tracing and metrics. I was thinking that before beginning to work on Logging API design, we could try to standardize the translation from Span and Metric events into structured logging events. This was requested as a feature for the metric SDK (open-telemetry/opentelemetry-specification#617), and I think the same could be said of spans. Right now many OTel repos are including standard-output exporters for both trace and metrics, but there are no standard conventional data formats for them to use, so it's a bit arbitrary what you get. I think it would be great if we had a standard structured log form for both metrics and spans.
    I would add that the OTel metrics API was conceived as a logging API from the start. The data model describes metric events, which can be recorded as a faithful way to capture metrics, so we would just need to standardize how they are structured as log records. Here's the event structure: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/metrics/api.md#metric-event-format
    Tigran Najaryan
    @tigrannajaryan
    @jmacd do you mean a standardized way to encapsulate span data and metric data inside a log record or something else?
    Joshua MacDonald
    @jmacd
    yes
    say you wanted to simply write spans into elastic search, for example
    Tigran Najaryan
    @tigrannajaryan
    Got it. Do you want to do an OTEP? (Assuming you want to get distracted from metrics :-) )
    Joshua MacDonald
    @jmacd
    I don't. I can't :)
    Ryan-Git
    @Ryan-Git
    Hi all. I just come up with an idea. As we will transfer logs to a remote collector in some sense, could we leverage dictionary encoding? Just like nano log(https://www.usenix.org/conference/atc18/presentation/yang-stephen). Briefly it makes a dictionary for static parts of every log message and writes only dynamic part as a compact binary log at runtime, which is super fast. This binary form is easier to transfer and analyze. A post processor is needed to make it human readable.
    8 replies
    Omer Katz
    @thedrow
    @jmacd I think you're on the right path
    51 replies
    Mats Taraldsvik
    @meastp

    Hi - I'd sometimes like to upload data dumps / full request dumps etc. and couple this to a specific trace (event/attribute). I think this might belong in the logging camp. However, I beleive that the data dumps/blobs are too big to transport to the collector via exporters (??) What I was thinking, is to use a logging attribute e.g. 'dump.file_path' in the application and write the file to /tmp, and in the agent (collector on the same node) in a "custom" processor:

    1. look for 'dump.file_path'
      2 if the file exists:
      3 upload to a cloud service (e.g. Azure Blob Storage)
    2. replace 'dump.file_path' with 'dump.file_uri' from the cloud service

    Is this a sensible thing to do?

    3 replies
    David Poncelow
    @zenmoto
    Hi- I've got an OTEP PR going about adding correlation information to existing logs at https://github.com/open-telemetry/oteps/pull/114- I'd appreciate any feedback about it or approvals if it looks good.
    6 replies
    Tigran Najaryan
    @tigrannajaryan
    @/all please review Logs protocol PR: open-telemetry/opentelemetry-proto#151
    Avi Kessner
    @avik-so
    I'm working on the OTEP for storing metric and tracing data into the log format.
    Which formats should I be looking at besides OT, Zipkin and Jaeger?
    7 replies
    Tigran Najaryan
    @tigrannajaryan
    I will not be able to attend the SIG meeting today.
    Morgan McLean
    @mtwo
    me neither
    I know that a number of people are on vacation this week - are there any objections to cancelling?
    Tigran Najaryan
    @tigrannajaryan
    No objections. Most of Splunk is off today.
    Morgan McLean
    @mtwo
    ok I will cancel it
    and I'll ask the same in the collector Gitter
    Tristan Sloughter
    @tsloughter
    has there been any decisions/discussion on if logging will have a similar concept of providers to tracers and meters? will resources be shared across all 3?
    10 replies
    there is work being done on Erlang's (same used by Elixir) logging library and support for default metadata (resources in otel world) and I want to make sure it doesn't conflict with otel logs but can be used to accomplish the same thing
    Ron Murphy
    @murphyzlaw
    Hello to the group - At Walmart we're interested in picking up your evolving work and incorporating into our instrumentation strategy. I would like to do a prototype with a use case of ours that requires structured logs (with various simple object or built-in type data). We're interested in logging to a sidecar, and concerned for efficient serializations/throughput (not just JSON over the wire). Languages for the source data would be Java and Javascript. A lot of this seems to line up with what you're working on, but it does seem you're pretty early on, so I am wondering how we could track what you're doing on our end?
    10 replies
    Tigran Najaryan
    @tigrannajaryan
    @murphyzlaw it maybe worth attending the next Log SIG meeting. They are open for anyone to attend. You can also track progress in specification and collector repos (filter issues/PRs by "logs" label). FYI, @zenmoto also works on an OpenTelemtry-compliant Java logging PoC.
    Ron Murphy
    @murphyzlaw
    Thanks for the replies - will join this Wednesday. I have looked at most of the collateral you guys have produced via working documents, gitter, etc. Will take a look at fluent-logger-java and fluent-logger-node.
    Tigran Najaryan
    @tigrannajaryan
    Log SIG meeting tomorrow overlaps with OpenTelemetry Community meeting. I was out for a few days and don't have any updates since the last week. Does anyone have anything to discuss tomorrow? I can reschedule the meeting tomorrow after the Community meeting, but if there is nothing to discuss we can skip this time.
    Tigran Najaryan
    @tigrannajaryan
    Cancelling since I don't see any objections.
    Tigran Najaryan
    @tigrannajaryan
    I added an overview of how logging works with OpenTelemetry to the specs, please review: open-telemetry/opentelemetry-specification#737
    Dr. Anton Chuvakin
    @anton_chuvakin_twitter
    Thanks to whoever invited me here - this is a very fun group, to be sure.
    Greg Shriver
    @gshriver
    Hello to the group - sorry for joining the SIG meeting uninvited. I see now, after reading this chat, that one should be invited before joining. Apologies for the breach of etiquette. I work in the mainframe division at Broadcom, and we have been working on several items that seem to be congruent with OT. First, we have been socializing live streaming of log data from the mainframe to a central sink, combining it with log data from non-mainframe sources, to allow easier log analysis for multi-platform applications, portions of which run on the mainframe. One of our other use cases centers around OT support (in our APM product) for distributed tracing.
    Christian Beedgen
    @kumoroku
    hi Greg––welcome. and, no worries: you don't need an invite!
    Greg Shriver
    @gshriver
    Thanks Christian!
    Tigran Najaryan
    @tigrannajaryan
    @gshriver welcome. OpenTelemetry SIG meetings are open for attendance, there is no need for invitation.
    Tigran Najaryan
    @tigrannajaryan
    @/all a reminder, please review the Logs Overview document: open-telemetry/opentelemetry-specification#737
    It sets the stage for several things that we want to do.
    Daniel Jaglowski
    @djaglowski

    Hi - I’m implementing a logs receiver, and wish to call consumer.ConsumeLogs(ctx context.Context, ld pdata.Logs) error.

    I have a struct representing log records, and wish to convert this to pdata.Logs. However, I’m not finding a way to do this that doesn’t require direct interaction with internal packages.

    Am I right to think this should be possible? Is there an expected approach to this that someone could point me to?

    10 replies
    Tigran Najaryan
    @tigrannajaryan
    @/all there is going to be a demo of another log collection agent tomorrow in Collector's SIG meeting. I have no updates on logs for tomorrow and unless someone has updates or anything to discuss about logs I suggest if you are interested in the demo join the Collector SIG meeting and we cancel the Log SIG this time.
    2 replies
    Avi Kessner
    @avik-so
    What time is the sig meeting?
    Hi, I have a question about logging context data.
    I want all of my logs to include the relevant request-id or session id.
    I also want to be able to log an event from deep within the call stack.
    I also don't want to pass my request or context object into every single function call.
    How is this problem normally solved in async /single-threaded systems like nodejs?
    Tigran Najaryan
    @tigrannajaryan
    Meeting times are in OpenTelemetry calendar: https://github.com/open-telemetry/community#calendar
    1 reply
    You may want to ask the nodejs question in nodejs channel. Logs are no different from traces and metrics from the perspective of context passing and nodejs SIG likely already thought about this.
    2 replies
    Tigran Najaryan
    @tigrannajaryan
    Please review Logs 1.0 GA scope / todo list: open-telemetry/oteps#130
    Andre Guerlain
    @aguerlain-r7
    Is the call ongoing? I don't see anyone in the LOG SIG Zoom
    Tigran Najaryan
    @tigrannajaryan
    Log SIG meeting is at 3pm PT this week. Check OpenTelemetry calendar.