These are chat archives for juttle/juttle

5th
Jan 2016
Michael Demmer
@demmer
Jan 05 2016 01:00
@rlgomes the link you sent shows a passing build
Rodney Lopes Gomes
@rlgomes
Jan 05 2016 01:01
because I reran and forgot it actually obliterates the existing run :(
Michael Demmer
@demmer
Jan 05 2016 01:01
indeed
Rodney Lopes Gomes
@rlgomes
Jan 05 2016 01:01
this would be a good moment for me to start swearing on our public channel but i'll be nice
Michael Demmer
@demmer
Jan 05 2016 15:06
@dmajda re your comment on juttle/juttle@4819515
As part of excising moment-parser, @welch proposed that we remove support from the moment grammar for expressions like “last 3 months”.
The reasoning was that the phrase “last 3 months” actually sounds like an interval and not a moment
we still support “last month”, “this month”, and “next month”, just not “last/next <N> months"
Michael Demmer
@demmer
Jan 05 2016 15:11
One thing I noticed in this discussion is that we seem to no longer support “day 1 of 3 months ago”. That seems like a nice to have though again that wasn’t documented, even though it was supported.
David Majda
@dmajda
Jan 05 2016 15:12
OK, that change is fine with me.
Michael Demmer
@demmer
Jan 05 2016 15:13
Cool.

Also as per your comment about unifying MomentConstant and MomentLiteral, I agree that seems more sensible and I actually started down that road originally.

I abandoned it because it seems irreconcilable with the goal of not having juttle know anything about the internal AST structure used by moment-parser.

The reason is that as part of the flowgraph phase of compilation, we need to take moment literal expressions and turn them into JS objects (we currently use JuttleMoment for this) so we can do compile-time arithmetic etc on the objects. But then we need to turn this back into a data structure (we currently use MomentConstant for this) to be part of the flowgraph.
So my thought was that in fact we do want separate things for MomentLiteral and MomentConstant — the former represents some expression that moment-parser can turn into a moment for us, while the second is always just an ISO timestamp and therefore we can directly create a JuttleMoment out of it.
David Majda
@dmajda
Jan 05 2016 15:33
Well, right now it is MomentLiteral from parser to the semantic pass, MomentConstant from there onward (similarly for durations). My idea is to move MomentConstant creation to the parser, which will eliminate the need for MomentLiteral completely (and also avoid requiring moment-parser at two places in the code, which I find slightly odd). Once this is done, I'd rename MomentConstantto MomentLiteral for consistency with other literals. This should be it.
Michael Demmer
@demmer
Jan 05 2016 15:35

Oh I see. That makes sense.

It would require passing the values for “now”, “beginning”, and “end” into the parser, which seems slightly odd but I guess it’s reasonable.

David Majda
@dmajda
Jan 05 2016 15:37
Yeah. I think we really need to pass just now, the rest is consts.
Michael Demmer
@demmer
Jan 05 2016 15:37
Oh yes. True.
OK — although I’m somewhat reluctant to put more time into this, I think this should be a simple enough change that I’m on board with doing it as part of the PR. I’ll see if I can whip this up quickly enough.
David Majda
@dmajda
Jan 05 2016 15:39
OK. As a historical side note, I wanted to do all this few months ago already. The multitude of moment-related AST nodes really bothered me, but I managed to excise them only from later compiler stages (the absence of decent compiler interface stopped me from going further -- this is a non-issue now).
Michael Demmer
@demmer
Jan 05 2016 15:40
Good to know.
Michael Demmer
@demmer
Jan 05 2016 15:58
As predicted that was easy enough — @dmajda please take a look at juttle/juttle@a326f16
Just pushed another fix for the test — both of these would squash with the earlier history assuming we feel good about it.
David Majda
@dmajda
Jan 05 2016 16:00
OK, looking.
Michael Demmer
@demmer
Jan 05 2016 16:00
Cool — will be afk for a bit but I can take your comments in a while.
David Majda
@dmajda
Jan 05 2016 16:20
OK, reviewed.
Oleg Seletsky
@go-oleg
Jan 05 2016 16:59
I put together a juttle flowgraph visualizer POC that shows points flowing through the flowgraph of a running juttle program and lets you go back in time and “replay” step by step: https://github.com/juttle/juttle-flowgraph-viz. Its not necessarily on the roadmap, but feedback welcome.
Balázs Kutil
@bkutil
Jan 05 2016 17:13
@go-oleg cool! how hard would it be to show ticks & marks as well? I think that'd be really useful for debugging...
Daria Mehra
@dmehra
Jan 05 2016 17:19
@go-oleg tried but got this from gulp run
Error: Cannot find module 'redux-thunk' from '/Users/dmehra/git/juttle-flowgraph-viz/src'
Oleg Seletsky
@go-oleg
Jan 05 2016 17:28
@bkutil, yea that should be straightforward. created juttle/juttle-flowgraph-viz#1
@dmehra, pushed a fix for that (thats what i get for having npm modules installed in my home dir…and not adding tests for now)
Daria Mehra
@dmehra
Jan 05 2016 17:48
@go-oleg, some more:
Error: Cannot find module 'd3' from '/Users/dmehra/git/juttle-flowgraph-viz/src/components’
Error: Cannot find module 'jquery' from '/Users/dmehra/git/juttle-flowgraph-viz/node_modules/backbone'
Matthew Nibecker
@mnibecker
Jan 05 2016 17:56
somebody is using backbone
Oleg Seletsky
@go-oleg
Jan 05 2016 17:57
yea juttle…will file an issue so we can get this resolved for other juttle consumers
Michael Demmer
@demmer
Jan 05 2016 19:18
I think the only use of backbone in juttle was for the events stuff and there’s already an issue for that.
Oleg Seletsky
@go-oleg
Jan 05 2016 19:25
ah right, juttle/juttle#49
Matthew Nibecker
@mnibecker
Jan 05 2016 19:37
I created an issue for my proposed http read -listenfunctionality. Would appreciate comments/hate.
juttle/juttle#111
Rodney Lopes Gomes
@rlgomes
Jan 05 2016 19:37
I had the same thought on the bart yesterday going home! so +100!
Matthew Nibecker
@mnibecker
Jan 05 2016 19:39
well it was actually @demmer ’s idea to a complaint I had
Rodney Lopes Gomes
@rlgomes
Jan 05 2016 19:40
I think the http read name though will be confusing as hell given we already have read http
maybe listen http would better express what this does: ie creates a listener for http requests
Mark Stemm
@mstemm
Jan 05 2016 19:41
it has to be read http right?
Rodney Lopes Gomes
@rlgomes
Jan 05 2016 19:41
and gives us room to create things like listen tcp listen graphite ... etc. to listen to other types of incoming requests ?
Mark Stemm
@mstemm
Jan 05 2016 19:42
or are you proposing a new thing that’s alongside read and write
Matthew Nibecker
@mnibecker
Jan 05 2016 19:42
http read is actually typo, meant read http, but I like your idea
Rodney Lopes Gomes
@rlgomes
Jan 05 2016 19:42
I think we'd want to distinguish clearly that this source sticks aroudn forever and doesn't have the -from/-to associated with it
so yes I'm suggesting this isn't an adapater but something else maybe we could call listener, connector (please not receiver ;) )
Daria Mehra
@dmehra
Jan 05 2016 19:44
@go-oleg i’m now successfully running the visualizer, but given this program, it just shows 0 in | 0 out for points:
(
emit -points [ { user: "u1", dept: "d1" }, {user: "u2", dept: "d2"}, {user: "u3", dept: "d3"} ];
emit -points [ { dept: "d1", mgr: "m1" }, { dept: "d3", mgr: "m3" }, {dept: "d4", mgr: "m4" } ];
)
| join -outer 1 dept
worked that out, it needs explicit sink at the end.
very cool tool
Rodney Lopes Gomes
@rlgomes
Jan 05 2016 19:51
@demmer you had the original idea of creating the http server don't you think it also opens the world to a new thing that isn't an adapter and therefore we'd distinguish by not using the read verb proceeding it and instead opt for listen (or anything else as long as it clearly distinguishes that this is not actively reading but insteading waiting for something to be sent by an external source)
Daria Mehra
@dmehra
Jan 05 2016 20:34
listen vs read makes intuitive sense to me as a user
is the logic then, in addition to the current “this adapter can read and write”, to end up with “this adapter can read and write and listen” (and this other adapter doesn’t support “listen”)?
Daria Mehra
@dmehra
Jan 05 2016 20:40
which adapters other than http might have a listen mode?
also what about something like kafka, when we support it, is that a “read” or a “listen”?
Rodney Lopes Gomes
@rlgomes
Jan 05 2016 20:54
listen is for receiving data on an actively listening source. That means you could do listen http to listen for http traffic or possibly listen graphite to listen for graphite data in the well known carbon format. I just feel that doing read http -serverMode true isn't really going to read anything on its own.
Daria Mehra
@dmehra
Jan 05 2016 21:52
we have a project wiki: https://github.com/juttle/juttle/wiki. since there's no PR workflow for this, please look at it and edit whatever needs editing in place. because wiki.
i considered making a custom sidebar for the wiki, to have some organization on the right, but i think doing that will remove the auto-generated one that lists all the pages, so then adding a new page will require you to add it to the sidebar explicitly, otherwise nobody would ever find it.
so, keeping the auto-gen alphabetical listing on the right for now.
if anyone wants to get fancy, the wiki supports headers and footers.
Michael Demmer
@demmer
Jan 05 2016 22:19
@rlgomes et al First of all I think original credit for read http -listen is due to @go-oleg. I think I had shamelessly forgotten that it was his idea when I mentioned it to @mnibecker
I strongly disagree with the idea of adding a listen proc to the language — it seems like it makes things more cluttered.
Also from the standpoint of juttle, you’re still reading points into the flowgraph — it’s just that you’re reading them in response to someone pushing to you as opposed to fetching them from somewhere else.