These are chat archives for juttle/juttle

26th
Jan 2016
David Cook
@davidbcook
Jan 26 2016 19:43
Hey everyone, if I have data streaming to my computer over a websocket, how would you recommend I connect that stream to juttle to visualize it?
Matthew Nibecker
@mnibecker
Jan 26 2016 19:45
Hey David, the data is from an outrigger or just a random web socket?
David Cook
@davidbcook
Jan 26 2016 19:45
random websocket
Matthew Nibecker
@mnibecker
Jan 26 2016 19:47
I think you would need an adapter for this. We recently created an http_server adapter in juttle so you could write a script to post points to that
Out to lunch will be back soon
David Cook
@davidbcook
Jan 26 2016 19:52
So write a script on my local computer to take the incoming points from the websocket, batch them up, then post them to the http_server? Seems like it would add a bit of latency. Didn't the old front-end use websockets?
The setup I'm working with now is actually pretty similar to the way jut used to work. We have a server in the cloud that accepts and stores data then forwards it on to clients via websocket.
Daria Mehra
@dmehra
Jan 26 2016 19:55
we still use websockets, the difficulty is in getting outrigger to connect to your websocket instead of spawning its own
it you want to use outrigger unchanged, you’d have to redirect the data from your websocket to something we understand, such as http_server
but you could fork outrigger :)
David Cook
@davidbcook
Jan 26 2016 19:56
ah I naively thought it would be simple to just pass the websocket details to outrigger upon startup
Daria Mehra
@dmehra
Jan 26 2016 19:57
or you could use visualizations directly via the new juttle-client-library
but it would take a bit of work on your end, and the APIs haven’t settled yet, so i wouldn’t send you down that path.
David Cook
@davidbcook
Jan 26 2016 19:59
ok
Daria Mehra
@dmehra
Jan 26 2016 19:59
as for passing websocket details to outrigger, let’s wait for @mnibecker to get back online, maybe there’s something he could do.
David Cook
@davidbcook
Jan 26 2016 19:59
ok
in the meantime, I really miss explorer. I tried to give myself some time to get used to the new juttle dev style, but it's just so much more involved. Making simple tweaks to the juttle code takes much longer and then you lose track of which outrigger tab corresponds to which file or which version of a specific file
Daria Mehra
@dmehra
Jan 26 2016 20:05
right, i think you’re looking for juttle/outrigger#30
my personal experience has been similar - it’s good to have the juttles in github for when i’m ready to commit them, but for quick iteration on the code, i’d rather do it in-browser. hence, waiting for juttle/outrigger#30 to land.
David Cook
@davidbcook
Jan 26 2016 20:06
indeed!
that would make life much easier
Daria Mehra
@dmehra
Jan 26 2016 20:07
would it serve your use case if we had editing in the browser, but not saving? i.e. when you need to save the juttle, you hand-copy it to your git repo or whever you keep juttles, and save there.
there’s a separate issue juttle/outrigger#76 to have ability to browse files from outrigger.
David Cook
@davidbcook
Jan 26 2016 20:08
yes that would help; saving is a nice to have
Daria Mehra
@dmehra
Jan 26 2016 20:09
so you’d be able to browse/load your saved juttles (say, from your git repo clone), and you can edit / write new juttles in the browser.
David Cook
@davidbcook
Jan 26 2016 20:10
yep
Jason Wang
@jasonbw
Jan 26 2016 20:24
Hi all
Daria Mehra
@dmehra
Jan 26 2016 20:25
you’re in the right place @jasonbw
Jason Wang
@jasonbw
Jan 26 2016 20:25
@dmehra - can you give me an example of reading from a local file?
Daria Mehra
@dmehra
Jan 26 2016 20:25
sure, it’s just like this
read file -file/tmp/input.json'
we support json, json lines, csv and log files (with grok rule support)
Jason Wang
@jasonbw
Jan 26 2016 20:27
thanks
it also looks like relative paths work: 'input.json'
Daria Mehra
@dmehra
Jan 26 2016 20:30
they do, but i occasionally get confused “relative to what”, hence the habit of using absolute paths. (also, been playing with docker too much.)
Matthew Nibecker
@mnibecker
Jan 26 2016 20:31
hey @davidbcook re, really missing explorer- have you tried using the outrigger-client watch command for development?
David Cook
@davidbcook
Jan 26 2016 20:31
I haven't
Matthew Nibecker
@mnibecker
Jan 26 2016 20:32
be curious to hear your experience with it
David Cook
@davidbcook
Jan 26 2016 20:34
didn't know about it; since it's not documented in the outrigger README, I'm guessing it would solve only the saving and relaunching of outrigger problem. It wouldn't help tie visualizations to code. Is that right?
Matthew Nibecker
@mnibecker
Jan 26 2016 20:36
Not quite sure what “tie visualizations to code” means
are you talking about the notebook style editing experience?
David Cook
@davidbcook
Jan 26 2016 20:37
when I have an outrigger tab open, I have no idea which code that tab corresponds to. I can use chart titles to guess, but that only works if I change the chart title every time I save a file
Matthew Nibecker
@mnibecker
Jan 26 2016 20:38
oh yeah, we just added a text field that displays the current juttle. Is that not helpful?
David Cook
@davidbcook
Jan 26 2016 20:39
haven't seen that yet; I'll check it out
@mnibecker did you see my comment about websockets and outrigger?
Matthew Nibecker
@mnibecker
Jan 26 2016 20:43
reading
so you want to put points from a websocket into the flowgraph of a running program?
David Cook
@davidbcook
Jan 26 2016 20:45
totes
Matthew Nibecker
@mnibecker
Jan 26 2016 20:45
I would do the http_server thing
so the program would look like thus:
read http_server -port 3000
| write elasticsearch
and you could http post points to that endpoint
is this what you desire?
David Cook
@davidbcook
Jan 26 2016 20:48
is the write elasticsearch required or just an option? I want to go straight to a visualization
Matthew Nibecker
@mnibecker
Jan 26 2016 20:49
you could just go straight into a visualization
David Cook
@davidbcook
Jan 26 2016 20:49
I'd rather not have the extra moving part of an http_server that just batches and forwards incoming points. Is there any reason you're recommending that route over hooking straight to the webhook? Is the latter not possible?
Matthew Nibecker
@mnibecker
Jan 26 2016 20:49
I was just using this to illlustrate that you could write the points to a store and them back again much like we did with points from segment
I recommend it because we don’t have a websocket adapter at this point
what’s the websocket you’re trying to connect to?
David Cook
@davidbcook
Jan 26 2016 20:51
we're already storing them, so that's not necessary
Matthew Nibecker
@mnibecker
Jan 26 2016 20:51
where are you storing the data?
David Cook
@davidbcook
Jan 26 2016 20:51
the websocket is from our back end server that receives the data (i'm pretty sure it's https://orchestrate.io/ )
Matthew Nibecker
@mnibecker
Jan 26 2016 20:52
so you only want to read live data from the websocket?
you don’t care about historical?
David Cook
@davidbcook
Jan 26 2016 20:52
at this point, yes
live only
Mark Stemm
@mstemm
Jan 26 2016 20:53
so it sounds like we’d want a read ws (read from a web socket) adapter
David Cook
@davidbcook
Jan 26 2016 20:53
exactly
Matthew Nibecker
@mnibecker
Jan 26 2016 20:53
yeah
Mark Stemm
@mstemm
Jan 26 2016 20:53
that seems reasonable to do
that source doesn’t happen to support non-websocket data fetches?
David Cook
@davidbcook
Jan 26 2016 20:56
I believe it does, but I'm primarily interested in live data right now, which seems to lend itself to websockets (if I'm not mistaken)
Mark Stemm
@mstemm
Jan 26 2016 20:56
filed juttle/juttle#269
David Cook
@davidbcook
Jan 26 2016 21:03
thanks!
Mark Stemm
@mstemm
Jan 26 2016 21:04
until that’s written, you could get by using read http