Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    dsisnero
    @dsisnero
    arr = Arrow::Array.new([0,1,1,1,1,2,3,4,2,6,7])
    fun = Arrow::Function.find('unique') returns a function but I am not sure how to run it. I tried fun.execute(arr) but get TypeError: no implicit conversion of Arrow::UInt8Array into Array
    fun.arity
    Sutou Kouhei
    @kou
    fun.execute([Arrow::ArrayDatum.new(arr)]).value
    I hope that we improve the API until the next release!
    dsisnero
    @dsisnero
    thank you. I will try it out. AlsoIs it possible to use the compute functions on a Arrow::ChunkedArray as well as an array? I noticed that Arrow::Array has instance methods with some of the compute functions but ChunkedArray doesn't.
    Sutou Kouhei
    @kou
    Some of compute functions don't support ChunkedArray yet.
    We can use them by concatenating Arrow::Arrays in ChunkedArray as workaround. (This causes overhead.)
    Right way is implementing ChunkedArray support in Apache Arrow C++.
    We need to implement Arrow::ChunkedArray#combine or something to concatenate Arrow::Arrays.
    I'll do it until the next release.
    dsisnero
    @dsisnero
    I already had submitted a JIRA issue for ChunkedArray combine_chunks. I was trying to see if I could do a faster groupby in ruby using the compute kernels but it would be better if this done in Arrow C++. Thanks for all your work
    Sutou Kouhei
    @kou
    It'll be included in the next release: apache/arrow#9621
    dsisnero
    @dsisnero
    Thank you for adding combine_chunks and array.concatenate
    Henry Tseng
    @henrytseng
    has any one had success installing the gem through a docker image based on apk?
    dsisnero
    @dsisnero
    added some memoryview requests https://bugs.ruby-lang.org/issues/17832, https://bugs.ruby-lang.org/issues/17831, https://bugs.ruby-lang.org/issues/17851 among others - I am not too comfortable with C yet
    Sutou Kouhei
    @kou
    Could you file 17831 and 17832 to https://github.com/ruby/fiddle/ ?
    We can resolve them in Fiddle.
    dsisnero
    @dsisnero
    Sutou Kouhei
    @kou
    Thanks.
    dsisnero
    @dsisnero
    I see red-arrow-flight in the arrow/ruby repository but there is no gem for this.
    dsisnero
    @dsisnero
    Also - looking at the Glib classes there doesn't seem to be any bindings for Arrow::Scalar so cannot use compute function 'add' Arrow::Function.find('add')
    dsisnero
    @dsisnero

    I got the unique compute to work as you showed but can't get sum to work
    fun = Arrow::Function.find('sum')

    sum = fun.execute([Arrow::ArrayDatum.new(ar)]).value

    I get nil as the result where ar= Arrow::Array.new([0,1,1,1,1,2,3,4,2,6,7])
    Sutou Kouhei
    @kou
    We'll release red-arrow-flight in the next release.
    Sutou Kouhei
    @kou
    We should improve scalar support.
    sum should work with the scalar support improvement.
    dsisnero
    @dsisnero
    @kou Thank you for all your great work
    Sutou Kouhei
    @kou
    No problem. Thanks for your report.
    jaspreetsd902
    @jaspreetsd902
    Is there a way I can change the compression type? I'm assuming it uses Snappy by default, i'd like to test with other compression methods as well (e.g. gzip2, LZO, etc)
    Sutou Kouhei
    @kou
    Are you saying about Apache Parquet format? Or other format?