Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Eirik Tsarpalis
    @eiriktsarpalis
    /cc @mathias-brandewinder
    Nick Palladinos
    @palladin
    Hi @lucasvfventura it would be great to have something like Spark.ML
    If you have mbrace related questions feel free to ask us.
    Lucas V. F. Ventura
    @lucasvfventura
    Hey guys have you hear of Prajna? Also I'm just starting with MBrace, do you have something close to a distributed data frame?
    Nick Palladinos
    @palladin
    Yes we are aware of Prajna. I think the main difference is that our main primitive is the cloud { ... } builder
    and we build everything on top of it.
    About the dataframe... we don't have something similar but I know that Tomas did some similar work with deedle and mbrace https://github.com/BlueMountainCapital/Deedle.BigDemo
    but I'm not familiar with the details of his work.
    Eirik Tsarpalis
    @eiriktsarpalis
    I should add that the MBrace.Flow library defines distributed data structures similar to data frames. But they are part of the library, not baked into the runtime.
    Similarly, a distributed data frame could be defined as part of an ML library
    or perhaps deedle could be integrated with MBrace
    Lucas V. F. Ventura
    @lucasvfventura
    For sure... I'm just taking the time to look deeper into F#, Scala and Spark. I'l try to make a demo combining the R TypeProvider and MBrace to build multiple machine learning models in a distributed way. I think doing this I'll get a sense of the MBrace.
    Eirik Tsarpalis
    @eiriktsarpalis
    Ok, great
    let us know if you have any questions
    afaik some work on mbrace/deedle has already been done by Tomas Petricek, see https://github.com/BlueMountainCapital/Deedle.BigDemo
    Lucas V. F. Ventura
    @lucasvfventura
    Unfortunately I have limited time to work on this =/.. only after work
    I'll take a look at that
    Eirik Tsarpalis
    @eiriktsarpalis
    Sure, no worries :-)
    Gauthier Segay
    @smoothdeveloper
    I'd like to know what this would look like if I had to implement this in C#, just want a rough idea
      let SetupEverything () =
        let uplanFolder = ""
        let clusterPort = 11111
        let networkTimeout = TimeSpan.FromSeconds(5.0)
        cloud {
          do! Cloud.Logf "making archive from folder %s" uplanFolder
          let! archive = makePowerstackCloudArchive uplanFolder
          do! Cloud.Logf "installing archive on machines"
          do! installPowerstack archive
          let! machines = getMachineNames ()
          let clusterMachine = machines |> Seq.head
          do! Cloud.Logf "start cluster on %s:%i" clusterMachine clusterPort
          do! startClusterProcess clusterMachine
          let clusterReachable = NetworkChecker.CheckHost networkTimeout { HostName = clusterMachine ; Port = clusterPort }
          if not clusterReachable.ListensToSocket then
            do! Cloud.Logf "cluster %s:%i not reachable" clusterMachine clusterPort
          let! serversOutput = startServerProcesses clusterPort clusterMachine
          return clusterMachine, serversOutput
        }
    Eirik Tsarpalis
    @eiriktsarpalis
    you mean using MBrace.CSharp?
    it can be done, but obviously it will be a lot more cumbersome
    using .OnSuccess(x => .. ) all the time
    Avi Avni
    @AviAvni
    @eiriktsarpalis please open new channel for mbrace.aws
    Kostas Rontogiannis
    @krontogiannis
    @AviAvni done
    Avi Avni
    @AviAvni
    Thanks
    Avi Avni
    @AviAvni
    Hi please look on my new PR started adding graph processing to mbrace.core