Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

scipipe/scipipe

on writing workflows with scipipe

People

Repo info

Activity

Mar 28 15:55

samuell on develop

Add support for go modules (compare)
Mar 28 15:54

samuell on master

Add support for go modules (compare)
Mar 28 15:54

samuell closed #96
Mar 28 15:48

dwmunster opened #96
Mar 26 11:15

samuell edited #95
Mar 26 11:14

samuell opened #95
Mar 18 10:46

dwmunster commented #94
Mar 18 09:34

samuell commented #94
Mar 13 12:43

dwmunster opened #94
Jan 03 14:56

samuell closed #93
Jan 03 14:56

samuell on master

Fix #93: formatting error in au… Add video tutorial page in docs (compare)
Jan 03 14:56

samuell on develop

Add video tutorial page in docs (compare)
Jan 02 15:54

samuell on develop

Fix #93: formatting error in au… (compare)
Jan 02 13:39

samuell edited #93
Jan 02 13:39

samuell edited #93
Jan 02 13:39

samuell opened #93
Oct 09 2019 12:29

kerkomen commented #75
Oct 09 2019 12:29

kerkomen commented #75
Oct 09 2019 09:30

samuell commented #92
Oct 09 2019 00:15

kerkomen opened #92

Drayton Munster

@dwmunster

where the expensive part is the data generation, then the post-processing and analysis steps are fairly lightweight

Samuel Lampa

@samuell

I see

Drayton Munster

@dwmunster

scipipe has been incredibly useful for me though (and will be cited), it sure beats manually writing out all 21*11=231 files to feed into my analysis script

Samuel Lampa

@samuell

scipipe has been incredibly useful for me though

That's fantastic to hear, thank you!

it sure beats manually writing out all 21*11=231 files to feed into my analysis script

haha

Drayton Munster

@dwmunster

tags and substreams are a key part of that, but I do bump into things there

but we've talked about a number of those

Samuel Lampa

@samuell

Yes. The areas that need to be fixed before calling it 1.0 I think.

Drayton Munster

@dwmunster

some of it is just a matter of documentation, like the right way/place to add tags

but I'll get around to contributing some of that

Samuel Lampa

@samuell

I'm very thankful. Your contributions and feedback have been very very valuable already.

Drayton Munster

@dwmunster

Glad to help, it's a neat project and was a great excuse to finally learn Go

Samuel Lampa

@samuell

Drayton Munster

@dwmunster

I've got a design question for you. What would would be the best way to handle the following?

Start a server,
Submit jobs to it,
Shutdown server once finished

One option I've thought of was possibly having a socket be the output of 1, that socket file feeds into 2, feed a substream (a la StreamToSubStream) into 3 which then ignores the files and sends a close message to the socket.

some overlap here with #38

My particular usecase would be to use a library like https://ray.io/ to 1) distribute my simulation jobs and 2) avoid starting a new python interpreter 231 times

Samuel Lampa

@samuell

It sounds to potentially have overlap with the ideas we discussed in #53

(Though I have to re-read it to double check ...)

Drayton Munster

@dwmunster

that Executor interface might be one way to do it, so long as it was possible to use for only a particular Process

A custom process that starts/stops the server in it's Run method could also probably work

Samuel Lampa

@samuell

that Executor interface might be one way to do it, so long as it was possible to use for only a particular Process

Yes, I think this is important. This would be a requirement either way I think, e.g. in the long planned (but very slowly progressing) kubernetes support, as one would then need to be able to specify different containers to use for each process.

Yes, custom processes allow to do most things, though with more code.

Drayton Munster

@dwmunster

And having to remember

*to clean up after yourself

Samuel Lampa

@samuell

Heh, yes.

Drayton Munster

@dwmunster

As much fun as deadlocks are

Samuel Lampa

@samuell

Well, this is another area that could be improved ... catching the Ctrl+C cancelling event and doing more graceful shutdown and clean-up.

Drayton Munster

@dwmunster

@samuell what's the minimum go version you'd want to support?

PR incoming to add go modules support, extremely trivial since there's no dependencies

Samuel Lampa

@samuell

PR incoming to add go modules support, extremely trivial since there's no dependencies

Sounds great! I don't think support for older Go versions is very important right now, as the user base is very small at the moment.

I think there is already a hard requirement for Go 1.8+ or something, but I think it is great to keep up with the changes coming in Go now.

Drayton Munster

@dwmunster

Cool, I put 1.13 in there but we can always bump it up if we decide we want a newer feature later

Samuel Lampa

@samuell

:thumbsup:

Drayton Munster

@dwmunster

What would you think about disabling the windows tests for now? The configuration means that the PR can still run Windows tests, but for now they just are noise on the PR

Samuel Lampa

@samuell

Indeed, yes, better to inactivate for now

Drayton Munster

@dwmunster

As far as I can tell, I think the best way to do that is on your side, you can disable the webhook and we can re-enable it later

Samuel Lampa

@samuell

Aight

Drayton Munster

@dwmunster

https://github.com/scipipe/scipipe/settings/hooks

It's been a while since I've used appveyor, at least I think it's done with webhooks

Samuel Lampa

@samuell

I don't find it there, but I'll dig it up somehow ...

It was under "Installed GitHub Apps" ... Removed now

Drayton Munster

@dwmunster

:+1:

Drayton Munster

@dwmunster

What was the motivation for Check and CheckWithMsg to die (via os.Exit(1)) instead of bubbling the error back to the user?

from a practical point of view, it makes it hard to test something that I know should error. In particular, making sure the segfault from #94 doesn't occur and that it errors instead

Samuel Lampa

@samuell

from a practical point of view, it makes it hard to test something that I know should error. In particular, making sure the segfault from #94 doesn't occur and that it errors instead

I see.

The reason has been primarily to simplify the API, with the thinking that the errors that are sent to Check/WithMsg() would be of the kind that would require a restart of a workflow in any case.

I guess one reason for simplifying as much as possible is that SciPipe was from the start aimed a lot towards bioinformatics, which has had a lot of non-programming-savvy people (wet lab folks) dipping their toes into it, and as such you would have to simplify as much as you can.

Though not sure that will/should be the primary user group as we go forward.

Drayton Munster

@dwmunster

An option would be to use errors internally and have something at the top that catches them and dies

But that is a valid point

Samuel Lampa

@samuell

An option would be to use errors internally and have something at the top that catches them and dies

Sounds reasonable