These are chat archives for juttle/juttle

16th
Mar 2016
饶琛琳
@chenryn
Mar 16 2016 09:16
Is @rlgomes here?
Kirill
@infloop
Mar 16 2016 13:22
Hi! Is Juttle using Node.js streams under the hood?
I have been looking into the source for a while and seems not. Am I missing something?
饶琛琳
@chenryn
Mar 16 2016 14:22
can't juttle-adopter-elastic support multi-field like path.raw in logstash schema?
Daria Mehra
@dmehra
Mar 16 2016 14:27
Hi @chenryn we are just adding full support for nested fields throughout juttle, give it another couple of weeks. Specifically for elastic adapter @davidvgalbraith can comment, maybe there's something we can do straight away for your use case. We do support accessing analyzed fields via field.raw. Are you able to give more detail of what you're trying to do?
David Majda
@dmajda
Mar 16 2016 14:43
@infloop No, we don't use Node.js streams, flowgraph nodes just pass around points as objects using regular method calls. In fact, I wasn't aware that it's possible to have non-textual Node.js streams -- I only found out about the object mode now when I looked at the docs.
Kirill
@infloop
Mar 16 2016 14:48
@dmajda Ok. I am a bit surprised after a journey into sources (I was absolutely sure that streams are somewhere in the base classes) :)
@dmajda Do you have any thoughts/plans to use them?
@dmajda I really want to use Juttle for some heavy stream processing ))
David Majda
@dmajda
Mar 16 2016 14:52
Oh, right, they are used in limited way in reading from files, HTTP streams, etc. This is pretty much given by Node.js APIs which use them. But they are not used for sending points around the flowgraph.
TBH, I'm not sure what using Node.js streams there would get us (beside additional dependency), but @bkutil may want to have a look more deeply.
Kirill
@infloop
Mar 16 2016 14:58
In fact is is possible to manually emit points at any given time in Node.js streams. Right, there are some issues with native streams like buffering, but to be honest these things are very rare.
Ok, I will continue digging...
饶琛琳
@chenryn
Mar 16 2016 15:17
@dmajda I just try read elastic -index 'logstash-*' -from :1 year ago: -to :now: | reduce -every :week: count() by 'path.raw', I got the result, but there are 'no known property "path.raw"' warning...
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 15:17
@chenryn are you still experiencing the error in the browser ? "a.default.values(...).find is not a function" ?
饶琛琳
@chenryn
Mar 16 2016 15:18
@rlgomes yes!
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 15:19
ok well looking at the various versions of your os + node + browser those seem perfectly fine so it must be something else
can you see anything in the browser console logs ?
饶琛琳
@chenryn
Mar 16 2016 15:23
TypeError: Object.assign is not a function {stack: (...), message: "Object.assign is not a function"}
(anonymous function) @ index.js:25
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 15:24
Give me a sec I saw something like this but to do with Array methods that wer enot a function and it was chrome specific (ie some bug in an older version of chrome). Give me 2 minutes to figure out if I can repro in an older version
Daria Mehra
@dmehra
Mar 16 2016 15:25
@chenryn in your elastic program, that's the right approach; could you post a complete warning? It may be that path field is missing in some data points and juttle warns you about that
饶琛琳
@chenryn
Mar 16 2016 15:27
@rlgomes yes, I just open my safari and get the right page!(sorry to my never used safari:)
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 15:28
I know the latest chrome is now at 47 so 42 is quite old but I am going to see if there's some bug in chrome with a possible workaround we could implement to avoid this situation
if you upgrade your chrome you shouldn't have trouble or just use Safari right now and lets see if there's a way we can easily fix this for v42 of chrome.
Oleg Seletsky
@go-oleg
Mar 16 2016 15:28
@rlgomes, I think we need to throw https://babeljs.io/docs/plugins/transform-object-assign/ into our .babelrc.
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 15:30
the babelrc entry would pull the pollyfill in ? http://stackoverflow.com/questions/31764414/object-assign-is-not-a-function
@go-oleg ^ I can test this locally as I'm using docker to pull an older version of chrome to reproduce the issue
Oleg Seletsky
@go-oleg
Mar 16 2016 15:32
it’ll most likely change the code to use es5 functionality
饶琛琳
@chenryn
Mar 16 2016 15:32
@dmehra
Warning: index "logstash-2016.03.03" has no known property "path.raw" for type "default"
Warning: index "logstash-2016.03.03" has no known property "path.raw" for type "logs"
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 15:36
@chenryn will work on the issue with chrome v42 and report back a little later (I'll update the issue itself) but we should have a fix
Daria Mehra
@dmehra
Mar 16 2016 15:37
Ah, the logstash warning issue is juttle/juttle-elastic-adapter#126, fix is in the works and for now, please ignore the warnings, sorry about that.
They are meaningless and you are getting back correct data in the output.
饶琛琳
@chenryn
Mar 16 2016 15:42
3q for @rlgomes @dmehra . And one more question: can we do a hour-on-hour compare? I try (...yesterday;...today) | put hour=Date.get(time,"hour") | remove time | join hour, but seems time column can't be removed
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 15:49
@chenryn I think I know what you're trying to do but does the yesterday and today source have different fields ? as the join would make a single point for any given hour in which yesterday and today had the same time value. The other issue with converting the time the way you want I'm putting together a quick example
Daria Mehra
@dmehra
Mar 16 2016 15:50
whats the end goal of hour-by-hour comparison? If you want it overlaid on a timechart (yesterday's series vs today's) that's supported natively with -interval option to timechart view. If it's something else, explain a bit more?
饶琛琳
@chenryn
Mar 16 2016 15:53
ok, I know rlgomes' mean now. It's my wrong usage. And I haven't try chart view because the old chrome issue... thanks again...
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 16:12
@chenryn the timechart overlay should allow you to compare data from multiple days on the same timechart... let us know if that solves the issue or if you want to actually overlay the raw data to use for a different purpose. I'm working on an example of how to manipulate the time stamps from multiple days to line up like you were trying above
饶琛琳
@chenryn
Mar 16 2016 16:15
timechart can solve the issue, I'm glad to watch the new example too.
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 16:22
@chenryn took me a bit to get things to work nicely but here's an example of shifting/lining up date from multiple days so they can be joined or simply overlayed in different views:
sub shift_date(date) {
    put time = date + Duration.new(Date.get(time, 'hour')*60*60)
}

(
emit -from :2016-01-01: -every :hour: -limit 24
| put value1=10*Math.random()
| shift_date -date :today:;
emit -from :2016-02-01: -every :hour: -limit 24
| put value2=100*Math.random()
| shift_date -date :today:;
)
| join
the sub shift_date is where all the magic happens
now the above really only works for data that is lined up by the hour (let me think if there's a nicer way to handle any data by lining up based simply on milliseconds since the start of the day
Rodney Lopes Gomes
@rlgomes
Mar 16 2016 16:34
@chenryn now a solution where we can align your datas dates to a specific day
sub shift_date(date) {
    put time = date + (time - Date.startOf(time, 'day'))
}

(
emit -from :2016-01-01: -every :hour: -limit 24
| put value1=10*Math.random()
| shift_date -date :today:;

emit -from :2016-02-01: -every :hour: -limit 24
| put value2=100*Math.random()
| shift_date -date :today:;
)
| join
this new way means we simply subtract the current time field value from the midnight of that day figuring out exactly how far into the day that timestamp is and add that to the desired start of the day in the example above we're aligning to :today: which is basically midnight of today