Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Mar 28 15:55

    samuell on develop

    Add support for go modules (compare)

  • Mar 28 15:54

    samuell on master

    Add support for go modules (compare)

  • Mar 28 15:54
    samuell closed #96
  • Mar 28 15:48
    dwmunster opened #96
  • Mar 26 11:15
    samuell edited #95
  • Mar 26 11:14
    samuell opened #95
  • Mar 18 10:46
    dwmunster commented #94
  • Mar 18 09:34
    samuell commented #94
  • Mar 13 12:43
    dwmunster opened #94
  • Jan 03 14:56
    samuell closed #93
  • Jan 03 14:56

    samuell on master

    Fix #93: formatting error in au… Add video tutorial page in docs (compare)

  • Jan 03 14:56

    samuell on develop

    Add video tutorial page in docs (compare)

  • Jan 02 15:54

    samuell on develop

    Fix #93: formatting error in au… (compare)

  • Jan 02 13:39
    samuell edited #93
  • Jan 02 13:39
    samuell edited #93
  • Jan 02 13:39
    samuell opened #93
  • Oct 09 2019 12:29
    kerkomen commented #75
  • Oct 09 2019 12:29
    kerkomen commented #75
  • Oct 09 2019 09:30
    samuell commented #92
  • Oct 09 2019 00:15
    kerkomen opened #92
Drayton Munster
@dwmunster
where the expensive part is the data generation, then the post-processing and analysis steps are fairly lightweight
Samuel Lampa
@samuell
I see
Drayton Munster
@dwmunster
scipipe has been incredibly useful for me though (and will be cited), it sure beats manually writing out all 21*11=231 files to feed into my analysis script
Samuel Lampa
@samuell

scipipe has been incredibly useful for me though

That's fantastic to hear, thank you!

it sure beats manually writing out all 21*11=231 files to feed into my analysis script

haha

Drayton Munster
@dwmunster
tags and substreams are a key part of that, but I do bump into things there
but we've talked about a number of those
Samuel Lampa
@samuell
Yes. The areas that need to be fixed before calling it 1.0 I think.
Drayton Munster
@dwmunster
some of it is just a matter of documentation, like the right way/place to add tags
but I'll get around to contributing some of that
Samuel Lampa
@samuell
I'm very thankful. Your contributions and feedback have been very very valuable already.
Drayton Munster
@dwmunster
Glad to help, it's a neat project and was a great excuse to finally learn Go
Samuel Lampa
@samuell
:)
Drayton Munster
@dwmunster

I've got a design question for you. What would would be the best way to handle the following?

  1. Start a server,
  2. Submit jobs to it,
  3. Shutdown server once finished

One option I've thought of was possibly having a socket be the output of 1, that socket file feeds into 2, feed a substream (a la StreamToSubStream) into 3 which then ignores the files and sends a close message to the socket.

some overlap here with #38
My particular usecase would be to use a library like https://ray.io/ to 1) distribute my simulation jobs and 2) avoid starting a new python interpreter 231 times
Samuel Lampa
@samuell
It sounds to potentially have overlap with the ideas we discussed in #53
(Though I have to re-read it to double check ...)
Drayton Munster
@dwmunster
that Executor interface might be one way to do it, so long as it was possible to use for only a particular Process
A custom process that starts/stops the server in it's Run method could also probably work
Samuel Lampa
@samuell

that Executor interface might be one way to do it, so long as it was possible to use for only a particular Process

Yes, I think this is important. This would be a requirement either way I think, e.g. in the long planned (but very slowly progressing) kubernetes support, as one would then need to be able to specify different containers to use for each process.

Yes, custom processes allow to do most things, though with more code.
Drayton Munster
@dwmunster
And having to remember
*to clean up after yourself
Samuel Lampa
@samuell
Heh, yes.
Drayton Munster
@dwmunster
As much fun as deadlocks are
Samuel Lampa
@samuell
Well, this is another area that could be improved ... catching the Ctrl+C cancelling event and doing more graceful shutdown and clean-up.
Drayton Munster
@dwmunster
@samuell what's the minimum go version you'd want to support?
PR incoming to add go modules support, extremely trivial since there's no dependencies
Samuel Lampa
@samuell

PR incoming to add go modules support, extremely trivial since there's no dependencies

Sounds great! I don't think support for older Go versions is very important right now, as the user base is very small at the moment.

I think there is already a hard requirement for Go 1.8+ or something, but I think it is great to keep up with the changes coming in Go now.
Drayton Munster
@dwmunster
Cool, I put 1.13 in there but we can always bump it up if we decide we want a newer feature later
Samuel Lampa
@samuell
:thumbsup:
Drayton Munster
@dwmunster
What would you think about disabling the windows tests for now? The configuration means that the PR can still run Windows tests, but for now they just are noise on the PR
Samuel Lampa
@samuell
Indeed, yes, better to inactivate for now
Drayton Munster
@dwmunster
As far as I can tell, I think the best way to do that is on your side, you can disable the webhook and we can re-enable it later
Samuel Lampa
@samuell
Aight
It's been a while since I've used appveyor, at least I think it's done with webhooks
Samuel Lampa
@samuell
Selection_0101.png
I don't find it there, but I'll dig it up somehow ...
It was under "Installed GitHub Apps" ... Removed now
Drayton Munster
@dwmunster
:+1:
Drayton Munster
@dwmunster
What was the motivation for Check and CheckWithMsg to die (via os.Exit(1)) instead of bubbling the error back to the user?
from a practical point of view, it makes it hard to test something that I know should error. In particular, making sure the segfault from #94 doesn't occur and that it errors instead
Samuel Lampa
@samuell

from a practical point of view, it makes it hard to test something that I know should error. In particular, making sure the segfault from #94 doesn't occur and that it errors instead

I see.

The reason has been primarily to simplify the API, with the thinking that the errors that are sent to Check/WithMsg() would be of the kind that would require a restart of a workflow in any case.

I guess one reason for simplifying as much as possible is that SciPipe was from the start aimed a lot towards bioinformatics, which has had a lot of non-programming-savvy people (wet lab folks) dipping their toes into it, and as such you would have to simplify as much as you can.

Though not sure that will/should be the primary user group as we go forward.

Drayton Munster
@dwmunster
An option would be to use errors internally and have something at the top that catches them and dies
But that is a valid point
Samuel Lampa
@samuell

An option would be to use errors internally and have something at the top that catches them and dies

Sounds reasonable