Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Mark Stemm
    @mstemm
    We just released juttle-engine 0.4.0. It’s availble on:
    let us know if you have any problems or questions.
    Rodney Lopes Gomes
    @rlgomes
    @herki I merged a fix to master juttle/juttle#584 its missing documentation which I'll update shortly but basically you can do read file -file 'something.tsv' -separator '\t' and that would parse a tab separated value file (TSV).
    @herki let me know if you have any trouble the latest juttle master code and be aware the juttle master branch isn't comptable with the latest juttle-engine source. So we could cherry-pick the change into an official npm build for you if need that.
    Tomasz Muszynski
    @herki
    Ok I will check.
    Tomasz Muszynski
    @herki
    It's working perfect, I have csv file with separator ';' and every thing working now, thank you.
    Rodney Lopes Gomes
    @rlgomes
    Great!
    Kirill
    @infloop
    Hi! I am researching Juttle. Looks awesome. I have only one question: Can I process every point and do smth externally? I need to increment some counters in Redis and use the increment result? Something like: | put redis_write_command = 'HINCRBY juttle_hash value 1' | put value = write redis | view table
    Mark Stemm
    @mstemm
    Hi, thanks! Does ‘smth’ mean something?
    Kirill
    @infloop
    Edited the question.
    I know that I can't use put after sinks.
    But is it possible somehow?
    Mark Stemm
    @mstemm
    In order to do that, you’d want to fork the program into 2 branches. Something like the following:
    ... | (put redis_write_command = 'HINCRBY juttle_hash value 1' | write redis; view table)
    in one branch, you’re preparing the points to be written to redis. in the other branch, you’re viewing the output in a table
    Kirill
    @infloop
    Will check now.
    Mark Stemm
    @mstemm
    oh if you want the increment to be reflected in the table output, you’d have to do the increment in juttle instead of in the HINCRBY
    something like read redis | put value = value + 1 | (write redis; view table)
    Kirill
    @infloop
    Hmm... That's the problem that I need to increment in Redis...
    And somehow get the value from it back in the middle of the stream.
    Like native Node.js stream Transform.
    Michael Demmer
    @demmer

    Hi @infloop — unfortunately there really isn’t a good way to do this now.

    One idea would be to potentially extend the language so that in addition to read and write there would be a way for an adapter to expose an update implementation that would work as you suggest.

    But this isn’t something we can do right now.
    Kirill
    @infloop
    Oh, it is a pity :( I really love the idea of Juttle. Is it hard to extend the language for an adapter update? I mean I can try to do that.
    Mark Stemm
    @mstemm
    I suppose you could repeat the work of what the HINCRBY does. One thing that wouldn’t be covered is the side-effect stuff of creating the hash if doesn’t exist and initializing the field if it doesn’t exist. And I’m guessing you’re relying on that.
    Kirill
    @infloop
    The value I store in HINCRBY can be updated from different apps, even from different machines.
    So, I have to do that in Redis... I think @demmer offered a good solution.
    Mark Stemm
    @mstemm
    yeah so as @demmer said that kind of operation isn’t a good match for the capabilities of the adapters right now
    Kirill
    @infloop
    Yeah.
    Mark Stemm
    @mstemm
    regardless feel free to file a github issue for the feature request so we can keep track of it when looking for things to add
    Kirill
    @infloop
    I wonder is it hard for me to extend these capabilities?
    Hmm... I think I got the idea in sources. Will try to.
    Thanks.
    Michael Demmer
    @demmer
    Cool — feel free to shoot questions our way. I’m not 100% sure this is the best way to accomplish what you’d want, but there have been other cases where we’ve wanted to do similar things.
    饶琛琳
    @chenryn
    Is @rlgomes here?
    Kirill
    @infloop
    Hi! Is Juttle using Node.js streams under the hood?
    I have been looking into the source for a while and seems not. Am I missing something?
    饶琛琳
    @chenryn
    can't juttle-adopter-elastic support multi-field like path.raw in logstash schema?
    Daria Mehra
    @dmehra
    Hi @chenryn we are just adding full support for nested fields throughout juttle, give it another couple of weeks. Specifically for elastic adapter @davidvgalbraith can comment, maybe there's something we can do straight away for your use case. We do support accessing analyzed fields via field.raw. Are you able to give more detail of what you're trying to do?
    David Majda
    @dmajda
    @infloop No, we don't use Node.js streams, flowgraph nodes just pass around points as objects using regular method calls. In fact, I wasn't aware that it's possible to have non-textual Node.js streams -- I only found out about the object mode now when I looked at the docs.
    Kirill
    @infloop
    @dmajda Ok. I am a bit surprised after a journey into sources (I was absolutely sure that streams are somewhere in the base classes) :)
    @dmajda Do you have any thoughts/plans to use them?
    @dmajda I really want to use Juttle for some heavy stream processing ))
    David Majda
    @dmajda
    Oh, right, they are used in limited way in reading from files, HTTP streams, etc. This is pretty much given by Node.js APIs which use them. But they are not used for sending points around the flowgraph.
    TBH, I'm not sure what using Node.js streams there would get us (beside additional dependency), but @bkutil may want to have a look more deeply.
    Kirill
    @infloop
    In fact is is possible to manually emit points at any given time in Node.js streams. Right, there are some issues with native streams like buffering, but to be honest these things are very rare.
    Ok, I will continue digging...