Hi guys ) hope weekend is going fine. I have a question - is it possible to create a job with updatable input parameter, e.g. list or seq which is updated in real time and data is continuously received, like in observable. And then, we should continuously get results from the cloud computation as they become ready... it's like when the connection between the cloud runtime and client is open, so client is able to add input data and clouds can return updated results
let ch = runtime.DefaultStoreClient.Channel
let spA, rpA = ch.Create<int>()
let spB, rpB = ch.Create<int>()
let workflow (rpA : IReceivePort<int>) (spB : ISendPort<int>) =
cloud {
while true do
printfn "Waiting"
let! result = Cloud.Catch <| rpA.Receive(timeout = 1000)
match result with
| Choice1Of2 x -> do! spB.Send(x * x)
| Choice2Of2 _ -> ()
}
let job = runtime.CreateProcess(workflow rpA spB)
// Send from client
ch.Send(spA, 3)
// Receive in client
ch.Receive(rpB)
// Kill and dispose
job.Kill()
ch.Delete(rpA)
ch.Delete(rpB)
Hey guys, I'm getting started with F# and MBrace and I saw that you are looking for contributors to build a machine learning platform on top of MBrace, I would love participate in this project. Who should I contact? I have experience using R, Python, h2o and Spark, they all lack in some point and I think F# + MBrace could really be THE distributed ML platform. I have some ideas regarding model factory as well. Thanks
/cc @mathias-brandewinder
If you have mbrace related questions feel free to ask us.
and we build everything on top of it.
About the dataframe... we don't have something similar but I know that Tomas did some similar work with deedle and mbrace https://github.com/BlueMountainCapital/Deedle.BigDemo
but I'm not familiar with the details of his work.
Similarly, a distributed data frame could be defined as part of an ML library
or perhaps deedle could be integrated with MBrace
let us know if you have any questions
afaik some work on mbrace/deedle has already been done by Tomas Petricek, see https://github.com/BlueMountainCapital/Deedle.BigDemo
I'll take a look at that
I'd like to know what this would look like if I had to implement this in C#, just want a rough idea
let SetupEverything () =
let uplanFolder = ""
let clusterPort = 11111
let networkTimeout = TimeSpan.FromSeconds(5.0)
cloud {
do! Cloud.Logf "making archive from folder %s" uplanFolder
let! archive = makePowerstackCloudArchive uplanFolder
do! Cloud.Logf "installing archive on machines"
do! installPowerstack archive
let! machines = getMachineNames ()
let clusterMachine = machines |> Seq.head
do! Cloud.Logf "start cluster on %s:%i" clusterMachine clusterPort
do! startClusterProcess clusterMachine
let clusterReachable = NetworkChecker.CheckHost networkTimeout { HostName = clusterMachine ; Port = clusterPort }
if not clusterReachable.ListensToSocket then
do! Cloud.Logf "cluster %s:%i not reachable" clusterMachine clusterPort
let! serversOutput = startServerProcesses clusterPort clusterMachine
return clusterMachine, serversOutput
}
it can be done, but obviously it will be a lot more cumbersome
using .OnSuccess(x => .. ) all the time
