Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Marc Garcia
    @datapythonista
    No worries, you're always welcome
    Thanks for joining
    Matthew Turner
    @matthewmturner
    is it expected that the isort precommit hook would change setup.cfg (removing impala from isort known third parties)
    Marc Garcia
    @datapythonista
    Not sure, I don't use the precommit hook, but doesn't sound reasonable
    Marc Garcia
    @datapythonista
    @matthewmturner I reviewed your PR, I commented there why of this behaviour
    Matthew Turner
    @matthewmturner
    @datapythonista thanks! adding responses there
    Marc Garcia
    @datapythonista
    Thanks to everybody who participated in the sprint. Lots of great contributions, we have plenty of PRs ready to merge once the problem in the CI is fixed (we're almost there)
    Matthew Turner
    @matthewmturner
    thanks for your time and guidance! great experience.
    Edoardo Abati
    @EdAbati_gitlab
    Thank you @datapythonista for organising!
    Manik Kaushal
    @manikkaushal_gitlab
    Hi guyzz... Can anyone please help me in answering few questions on ibis performance??
    Daniel Kim
    @pybokeh
    Is there a way to access a table cell value? I am trying to parse the last name from a full name column containing string of the form: "last_name,first name" and so I have this code: associates['NAME'].mutate(LAST_NAME=associates['NAME'].split(',')[0]) But get no results since LAST_NAME column is of type ArrayColumn and so indexing does not apply. If I remove the [0] from the end of my split, I get a ArrayColumn, but there doesn't appear to be a way to access the scalar value or the cell value for each row in that column.
    Scott Hajek
    @scottcode
    @pybokeh I’m not sure about indexing into an ArrayColumn, but for your specific example sounds like re_extract might work:
    https://ibis-project.org/docs/generated/ibis.expr.api.StringValue.re_extract.html#ibis.expr.api.StringValue.re_extract
    Daniel Kim
    @pybokeh
    @scottcode That'll do even though I usually try to avoid regex since I often have to google or look-up the regex pattern. Thanks!
    jdb
    @jdb
    Hello, do you know what is the truth for the BigQuery SQL dialect understood by Ibis? Google exposes the parser for the "BigQuery Standard SQL" available under the project ZetaSQL (https://github.com/google/zetasql). Does Ibis uses this? Do you see benefits for Ibis to use this parser?
    For more context, I am trying figure how hard is it to add the BigQuery geospatial functions or the ML/AI functions to Ibis.
    Tim Swast
    @tswast
    The BigQuery connector does use Standard SQL syntax
    It is not currently relying on the ZetaSQL project itself (as far as I know there's not a Python wrapper for it yet)
    Adding functions is pretty straightforward. There is a guide here: https://ibis-project.org/docs/user_guide/extending/extending_elementwise_expr.html
    Daniel Kim
    @pybokeh
    Hello, I've searched in the API docs, but can't seem to find an equivalent cumcount() function that is available in pandas. In pandas, I would use cumcount() to create a ROW_NUM column, an example pandas code:
    df['ROW_NUM'] = df.sort_values(by=['PART_NUM']).groupby(['CLAIM_NUM']).cumcount() + 1 that is equivalent to performing SQL's ROW_NUMBER PARTION BY syntax, where I can then filter to the first record in each grouping (ROW_NUMBER ==1).
    Daniel Kim
    @pybokeh
    I did an inelegant work around which was to create a dummy column containing just 1s, then doing a cumsum() on that dummy column using ibis window function. I see there is an ibis.row_number, but not sure how to use that in my example.
    Daniel Kim
    @pybokeh
    I just realized that I can't user ibis.row_number() with a pandas backend. Using a Postgres backend, the ibis.row_number().over(w) does what I need or negates the need for cumcount(). That is pretty neat!
    Arsen Kairkanov
    @karsen_gitlab
    Hi all! I can't access the documentation at https://ibis-project.org/docs/ - always get 404. Is it a known issue? (Can't access since yesterday)
    Jeff Reback
    @jreback
    where do u see the other link?
    Arsen Kairkanov
    @karsen_gitlab

    http://ibis-project.org is working

    this page works for me too. At the top of this page you can see the links to Getting started, Documentation, Community...
    So the link to Documentation is not working for me.
    And my bookmarks of ibis documentation also not accessible with 404 error (e.g. https://ibis-project.org/docs/backends/impala.html)

    Safraz
    @safrazRampersaud
    Hello! Very excited about Ibis! ... Just started reading the docs, is Snowflake or SQL Server on the runway to be supported as a backend?
    Jeff Reback
    @jreback
    welcome to have contributions from the community ; there is an almost complete SQL server - just needs some tlc
    Renato Marroquín
    @RenatoMarroquin_twitter

    cool! that is awesome! is there a link where we can see/use that impl?

    there is an almost complete SQL server

    also I've noticed that the mysql impl is marked as experimental in the github page, I guess some features are missing , but is there a list of features that it should implement in order to be considered as "complete"?
    maybe having a table with the features each backend implements/lacks would help to know what is in place and what needs to be worked on
    Jeff Reback
    @jreback
    the sql back ends are very similarl and are essentially feature complete
    experimental is really about having folks support it if there are issues - though that label can prob come off mysql
    Safraz
    @safrazRampersaud
    Thank you @jreback ! Happy to keep in contact and support where I can.
    Safraz
    @safrazRampersaud
    @jreback tlc shown on the ibis-mssql module. Just created a PR to coordinate the imports in that module to the ibis-framework core project.
    Sasha Ovsankin
    @SashaOv
    Hi, is it possible to hack on Ibis without installing miniconda, with just PIP?
    Jeff Reback
    @jreback
    it’s possible not convenience as u hav sri make sure all deps for testing
    1 reply
    Florian Castelain
    @floriancastelain
    Hello. Is Ibis (connected to impala) thread safe ?
    Tilo Buschmann
    @tilo_buschmann_aok_gitlab
    Hi everyone. How would I go about using Ibis with SAP Hana? Is there a recommended generic way to connect using sqlalchemy (specifically sqlalchemy-hana) or should I create a new backend?
    Scott Hajek
    @scottcode
    @tilo_buschmann_aok_gitlab I think you would have to create a backend for that. It would be nice if there were a way to have a generic sqlalchemy backend that would cover the basic functionality for any available sqlalchemy dialect. Then it would only be necessary to create new specific backends if you want to leverage any of the functionality that is peculiar to that DB. But last I checked that hasn’t been done
    Tilo Buschmann
    @tilo_buschmann_aok_gitlab
    ok, I'll see what I can do
    Tilo Buschmann
    @tilo_buschmann_aok_gitlab
    I assume you know about the incompatibility between sqlalchemy >= 1.4.0 and ibis up to 1.4.0?
    the Binary module (interface?) has been removed from sqlalchemy, I'm pinning sqlalchemy to < 1.4.0 in my conda installation
    Jeff Reback
    @jreback
    this was fixed on master and >=1.3 sqlalchemy works
    Tilo Buschmann
    @tilo_buschmann_aok_gitlab
    great! I'll try that soon
    Tim Swast
    @tswast
    Alternatively, I could probably make one in http://github.com/googleapis, but will need to go through some red tape to make that happen.
    Jeff Reback
    @jreback
    we can create it in ibis
    actually looks like marc already did
    Tim Swast
    @tswast
    Yep. Thanks!