Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    lesbroot
    @lesbroot
    what's the score here?
    Krisztián Szűcs
    @kszucs
    Welcome everybody! Do You have any proposals about developing splearn? Maybe components needed to port?
    @vchollati thanks for your contribution!
    Kent Shih
    @texib
    hi everybody
    D. Rajeev. Reddy
    @drreddy
    Hello Iam trying to include the sparkit package in my scala project can anyone guide me how to add the package in the packageDependencies of scala
    Krisztián Szűcs
    @kszucs
    Hi! Sorry for the late answer. As far as I know Yout cannot use sparkit-learn in a scala project. It works the other way, PySpark depends on the JVM.
    Mustafa
    @Elbehery
    Hey guyz .. I know this room is for contirbutors, but I am stuck and I need urgent help .. I would like to use DBSCAN in parallel. So I want to know if I can run the Scikit-Learn on top of Apache Spark using Sparkit-learn .. Do u have any idea how to do so ? :D
    Mustafa
    @Elbehery
    ImportError: cannot import name _get_unmangled_double_vector_rdd
    Mustafa
    @Elbehery
    lensacom/sparkit-learn#55
    cypherpunker
    @cypherpunker
    This message was deleted
    I have this issue: lensacom/sparkit-learn#62
    András Fülöp
    @fulibacsi
    Hi @cypherpunker, we responded to your issue.
    cypherpunker
    @cypherpunker
    @fulibacsi thank you very much!
    András Fülöp
    @fulibacsi
    @cypherpunker the page you linked is a 404 now
    cypherpunker
    @cypherpunker
    thanks @fulibacsi, I moved it to github’s issues: lensacom/sparkit-learn#63
    Still with the same issue, any idea of how to solve it?
    András Fülöp
    @fulibacsi
    @cypherpunker sorry for taking so long to reply
    yeah, i can see what could be the problem
    i'll look into it
    can you please tell me your sklearn's version?
    sulphide
    @sulphide
    anyone know if there is a builtin way to convert a dict to a pyspark.sql.Row
    András Fülöp
    @fulibacsi
    i think you can simply init pyspark.sql.Row with a dict.
    cypherpunker
    @cypherpunker
    @fulibacsi thanks for the help. My version is :
    In [40]:
    
    import sklearn
    
    ​
    
    sklearn.__version__
    
    Out[40]:
    
    ‘0.17.1
    aremirata
    @aremirata
    just want to ask why is it that nmf under decomposition library is not included in sparkit-learn? Only truncatedsvd is found here.
    Krisztián Szűcs
    @kszucs
    we didn't have the time to implement
    PRs are welcome
    aremirata
    @aremirata
    hi guys, how can we get the transpose of a SparseRDD or ArrayRDD
    It gets this error
    ValueError: all the input array dimensions except for the concatenation axis must match exactly
    aremirata
    @aremirata
    @all, do you have some experience how to get transpose of ArrayRDD or sparseRDD?
    aremirata
    @aremirata
    @kszucs , do you have some idea how to get transpose of ArrayRDD or SparseRDD?
    Krisztián Szűcs
    @kszucs
    Hey! Transposing would change the partitioning axis.
    Currently there is no easy way to get the transposes
    I'd suggest You to use dask instead https://github.com/dask/dask
    For advanced array operations
    Krisztián Szűcs
    @kszucs
    OTOH You can transpose every block rdd.map(lambda x: x.T) to get horizontally partitioned rdd
    aremirata
    @aremirata
    thanks @kszucs . another question, is there a way we can perform column indexing in sparkit-learn?
    Krisztián Szűcs
    @kszucs
    like x[:, 1:-1] ?
    aremirata
    @aremirata
    thanks @kszucs . Dask looks great!
    Rohit Raj
    @RohitRaj2017
    Hi I have a dataframe in followin format
    DataFrame[foo: struct<key:string,value:string>]
    i want to access only key or value .. Please suggest
    I am using spark 1.6.1 with pyspark
    Rohit Raj
    @RohitRaj2017
    is anyone active in this group
    Krisztián Szűcs
    @kszucs
    Hey
    I think your question is related to spark dataframes, not sparkit's structures.
    Or do you want to create DictRDD's from the dataframe above?
    Rishabh Kumar
    @rishabhkumar296
    hi. Is sparkit-learn still under development? I would love to contribute to the codebase.