Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Mark Stemm
    @mstemm
    yeah so as @demmer said that kind of operation isn’t a good match for the capabilities of the adapters right now
    Kirill
    @infloop
    Yeah.
    Mark Stemm
    @mstemm
    regardless feel free to file a github issue for the feature request so we can keep track of it when looking for things to add
    Kirill
    @infloop
    I wonder is it hard for me to extend these capabilities?
    Hmm... I think I got the idea in sources. Will try to.
    Thanks.
    Michael Demmer
    @demmer
    Cool — feel free to shoot questions our way. I’m not 100% sure this is the best way to accomplish what you’d want, but there have been other cases where we’ve wanted to do similar things.
    饶琛琳
    @chenryn
    Is @rlgomes here?
    Kirill
    @infloop
    Hi! Is Juttle using Node.js streams under the hood?
    I have been looking into the source for a while and seems not. Am I missing something?
    饶琛琳
    @chenryn
    can't juttle-adopter-elastic support multi-field like path.raw in logstash schema?
    Daria Mehra
    @dmehra
    Hi @chenryn we are just adding full support for nested fields throughout juttle, give it another couple of weeks. Specifically for elastic adapter @davidvgalbraith can comment, maybe there's something we can do straight away for your use case. We do support accessing analyzed fields via field.raw. Are you able to give more detail of what you're trying to do?
    David Majda
    @dmajda
    @infloop No, we don't use Node.js streams, flowgraph nodes just pass around points as objects using regular method calls. In fact, I wasn't aware that it's possible to have non-textual Node.js streams -- I only found out about the object mode now when I looked at the docs.
    Kirill
    @infloop
    @dmajda Ok. I am a bit surprised after a journey into sources (I was absolutely sure that streams are somewhere in the base classes) :)
    @dmajda Do you have any thoughts/plans to use them?
    @dmajda I really want to use Juttle for some heavy stream processing ))
    David Majda
    @dmajda
    Oh, right, they are used in limited way in reading from files, HTTP streams, etc. This is pretty much given by Node.js APIs which use them. But they are not used for sending points around the flowgraph.
    TBH, I'm not sure what using Node.js streams there would get us (beside additional dependency), but @bkutil may want to have a look more deeply.
    Kirill
    @infloop
    In fact is is possible to manually emit points at any given time in Node.js streams. Right, there are some issues with native streams like buffering, but to be honest these things are very rare.
    Ok, I will continue digging...
    饶琛琳
    @chenryn
    @dmajda I just try read elastic -index 'logstash-*' -from :1 year ago: -to :now: | reduce -every :week: count() by 'path.raw', I got the result, but there are 'no known property "path.raw"' warning...
    Rodney Lopes Gomes
    @rlgomes
    @chenryn are you still experiencing the error in the browser ? "a.default.values(...).find is not a function" ?
    饶琛琳
    @chenryn
    @rlgomes yes!
    Rodney Lopes Gomes
    @rlgomes
    ok well looking at the various versions of your os + node + browser those seem perfectly fine so it must be something else
    can you see anything in the browser console logs ?
    饶琛琳
    @chenryn
    TypeError: Object.assign is not a function {stack: (...), message: "Object.assign is not a function"}
    (anonymous function) @ index.js:25
    Rodney Lopes Gomes
    @rlgomes
    Give me a sec I saw something like this but to do with Array methods that wer enot a function and it was chrome specific (ie some bug in an older version of chrome). Give me 2 minutes to figure out if I can repro in an older version
    Daria Mehra
    @dmehra
    @chenryn in your elastic program, that's the right approach; could you post a complete warning? It may be that path field is missing in some data points and juttle warns you about that
    饶琛琳
    @chenryn
    @rlgomes yes, I just open my safari and get the right page!(sorry to my never used safari:)
    Rodney Lopes Gomes
    @rlgomes
    I know the latest chrome is now at 47 so 42 is quite old but I am going to see if there's some bug in chrome with a possible workaround we could implement to avoid this situation
    if you upgrade your chrome you shouldn't have trouble or just use Safari right now and lets see if there's a way we can easily fix this for v42 of chrome.
    Oleg Seletsky
    @go-oleg
    @rlgomes, I think we need to throw https://babeljs.io/docs/plugins/transform-object-assign/ into our .babelrc.
    Rodney Lopes Gomes
    @rlgomes
    the babelrc entry would pull the pollyfill in ? http://stackoverflow.com/questions/31764414/object-assign-is-not-a-function
    @go-oleg ^ I can test this locally as I'm using docker to pull an older version of chrome to reproduce the issue
    Oleg Seletsky
    @go-oleg
    it’ll most likely change the code to use es5 functionality
    饶琛琳
    @chenryn
    @dmehra
    Warning: index "logstash-2016.03.03" has no known property "path.raw" for type "default"
    Warning: index "logstash-2016.03.03" has no known property "path.raw" for type "logs"
    Rodney Lopes Gomes
    @rlgomes
    @chenryn will work on the issue with chrome v42 and report back a little later (I'll update the issue itself) but we should have a fix
    Daria Mehra
    @dmehra
    Ah, the logstash warning issue is juttle/juttle-elastic-adapter#126, fix is in the works and for now, please ignore the warnings, sorry about that.
    They are meaningless and you are getting back correct data in the output.
    饶琛琳
    @chenryn
    3q for @rlgomes @dmehra . And one more question: can we do a hour-on-hour compare? I try (...yesterday;...today) | put hour=Date.get(time,"hour") | remove time | join hour, but seems time column can't be removed
    Rodney Lopes Gomes
    @rlgomes
    @chenryn I think I know what you're trying to do but does the yesterday and today source have different fields ? as the join would make a single point for any given hour in which yesterday and today had the same time value. The other issue with converting the time the way you want I'm putting together a quick example
    Daria Mehra
    @dmehra
    whats the end goal of hour-by-hour comparison? If you want it overlaid on a timechart (yesterday's series vs today's) that's supported natively with -interval option to timechart view. If it's something else, explain a bit more?
    饶琛琳
    @chenryn
    ok, I know rlgomes' mean now. It's my wrong usage. And I haven't try chart view because the old chrome issue... thanks again...
    Rodney Lopes Gomes
    @rlgomes
    @chenryn the timechart overlay should allow you to compare data from multiple days on the same timechart... let us know if that solves the issue or if you want to actually overlay the raw data to use for a different purpose. I'm working on an example of how to manipulate the time stamps from multiple days to line up like you were trying above
    饶琛琳
    @chenryn
    timechart can solve the issue, I'm glad to watch the new example too.
    Rodney Lopes Gomes
    @rlgomes
    @chenryn took me a bit to get things to work nicely but here's an example of shifting/lining up date from multiple days so they can be joined or simply overlayed in different views:
    sub shift_date(date) {
        put time = date + Duration.new(Date.get(time, 'hour')*60*60)
    }
    
    (
    emit -from :2016-01-01: -every :hour: -limit 24
    | put value1=10*Math.random()
    | shift_date -date :today:;
    emit -from :2016-02-01: -every :hour: -limit 24
    | put value2=100*Math.random()
    | shift_date -date :today:;
    )
    | join
    the sub shift_date is where all the magic happens